Add rate limiting to AI APIs

Your AI invoice generator is live — and it just collapsed under its own success. Fifteen users hit the endpoint at once, the third-party LLM API throttled you mid-request, and now your customers see 429 errors instead of polished invoices. Sound familiar? When you build AI-powered APIs, the provider's rate limits are a hard ceiling you can't negotiate. But you can control how your own API behaves under that ceiling: by adding rate limiting to your AI APIs, you protect your costs, your users, and your provider quota — before it breaks.

The problem this lesson solves

Every AI API call costs money, consumes tokens, and counts against a strict quota. The most common pain points when you ignore rate limiting:

• Cost blowups — an infinite retry loop can burn hundreds of dollars in minutes
• Provider bans — hammering an endpoint too fast can get your API key suspended
• Degraded UX — one user's burst of requests starves everyone else
• Unpredictable failures — you get 429s at exactly the worst moment

You need a way to say "slow down" to your own API before the provider does it for you. That's where rate limiting comes in — it's the polite, deliberate throttle that keeps your AI API healthy under any load.

Core concept / mental model

Think of rate limiting as a turnstile at a stadium entrance. Each fan (request) needs a token to pass through. The turnstile only lets a fixed number of fans through per minute, regardless of how many queue up. If too many arrive, they wait in line or get turned away with a polite "come back later."

For AI APIs, the turnstile sits between your users and the LLM provider. It answers two questions:

1. Who can make a request? — by user ID, API key, IP address, or tenant
2. How many requests per time window? — e.g., 10 requests per minute, or 1,000 tokens per day

The most common algorithm is the token bucket: you have a bucket that fills at a steady rate (e.g., 1 token per second) and has a maximum capacity (e.g., 5 tokens). Each request removes one token. If the bucket is empty, the request is rejected or queued.

Pro tip: Rate limiting is not about blocking users — it's about smoothing demand so your AI backend stays responsive and within quota.

How it works step by step

Adding rate limiting to an AI API involves three layers:

1. Choose a storage backend — in-memory, Redis, or database (for distributed apps)
2. Define a rate limit policy — e.g., 100 requests per minute per user
3. Apply the policy in middleware — before the AI call is made

Here's the logical flow:

User Request → Rate Limit Check → (Within limit?) → Forward to AI Provider
                       │
                       └── No → Return 429 Too Many Requests

The check happens before any expensive work, so you never waste tokens on requests you're going to reject.

Choosing the right limit

Set your limits based on the provider's quota, not on what users want. If your LLM provider allows 60 requests per minute, set your own limit at 50 to leave headroom for bursts.

Hands-on walkthrough

Let's implement rate limiting for a simple AI endpoint using FastAPI and slowapi (a token-bucket library built on limits).

Step 1: Install dependencies

pip install fastapi slowapi

Step 2: Create the app with rate limiting

from fastapi import FastAPI, Request
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded

limiter = Limiter(key_func=get_remote_address)
app = FastAPI()
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)

# Mock AI call — replace with real OpenAI/Anthropic call
import time

def call_llm(prompt: str) -> str:
    time.sleep(0.5)  # simulate latency
    return f"Echo: {prompt}"

@app.post("/generate")
@limiter.limit("5/minute")
async def generate(request: Request, prompt: str):
    result = call_llm(prompt)
    return {"result": result}

Run with uvicorn main:app --reload. Now, if a user sends more than 5 requests in a minute, they get a 429 HTTP response with a message like "Rate limit exceeded: 5 per 1 minute."

Step 3: Test it

# test_rate_limit.py
import requests

url = "http://localhost:8000/generate"
for i in range(7):
    r = requests.post(url, params={"prompt": f"hello {i}"})
    print(r.status_code, r.text[:60])

Expected output:

200 {"result":"Echo: hello 0"}
200 {"result":"Echo: hello 1"}
200 {"result":"Echo: hello 2"}
200 {"result":"Echo: hello 3"}
200 {"result":"Echo: hello 4"}
429 {"detail":"Rate limit exceeded: 5 per 1 minute"}
429 {"detail":"Rate limit exceeded: 5 per 1 minute"}

Adding per‑user limits

Instead of IP‑based, limit by API key or user ID:

from slowapi.util import get_remote_address
from fastapi import Header, HTTPException

def get_user_key(request: Request, api_key: str = Header(...)) -> str:
    return api_key  # in practice, look up user from key

limiter = Limiter(key_func=get_user_key)
app.state.limiter = limiter

@app.post("/generate")
@limiter.limit("100/minute")
async def generate(request: Request, api_key: str = Header(...), prompt: str):
    # ...

Now each API key gets its own 100‑per‑minute budget.

Compare options / when to choose what

The verdict: start with in‑memory if you're on a single server. Move to Redis the moment you scale to multiple workers — otherwise your limits will be per‑process, and users can slip through by hitting different workers.

For token‑based rate limiting (e.g., limiting by tokens per day instead of requests per hour), you'll need a standalone service like limits directly or a specialized library.

Troubleshooting & edge cases

1. Limits are too strict (users get 429 when they shouldn't)

• Cause: You set a limit lower than the burst your app needs.
• Fix: Use a token bucket with burst capacity. In slowapi, you can use "10/second;100/minute" to allow short bursts but cap the long-term average.

@limiter.limit("10/second;100/minute")
async def endpoint(request: Request):
    ...

2. Redis connection error

• Symptom: Cannot connect to Redis during rate limit check.
• Fix: Rate limiting should fail open — if Redis is down, let the request through rather than blocking everyone. Wrap the check in a try/except and log the error.

3. Limits not applied in multi‑worker app

• Cause: Using in‑memory limits on a load‑balanced deployment.
• Fix: Switch to Redis storage by passing a storage_uri to Limiter.

limiter = Limiter(key_func=get_remote_address, storage_uri="redis://localhost:6379")

4. Burst traffic from a single user

• Symptom: One user monopolizes the AI quota, others get slow response.
• Fix: Implement a concurrency limiter (max simultaneous requests per user) on top of rate limiting. Use limits with a semaphore or a separate middleware.

5. 429s caused by your own provider, not your API

• Symptom: Your API returns 500 because the LLM provider throttled you.
• Fix: Add retry with exponential backoff on provider calls, and set your own rate limit below the provider's to prevent hitting their ceiling.

import time
from openai import OpenAI
client = OpenAI()

def call_with_retry(prompt, max_retries=3):
    for attempt in range(max_retries):
        try:
            return client.completions.create(model="gpt-3.5-turbo", prompt=prompt)
        except Exception as e:
            if "429" in str(e) and attempt < max_retries - 1:
                time.sleep(2 ** attempt)
            else:
                raise

What you learned & what's next

You now know how to add rate limiting to AI APIs to control cost and protect your backend from overload. You can:

• Explain why rate limiting matters for AI APIs (cost, quota, reliability)
• Apply a token-bucket algorithm with slowapi in FastAPI
• Choose the right storage backend (in‑memory vs Redis)
• Troubleshoot common rate-limit issues

Next, you'll build on this foundation by exploring concurrency control and queuing — how to handle high‑traffic loads gracefully while your AI provider processes requests. You'll learn to implement request queues, parallel batching, and timeout strategies that complement the rate limits you just set up.