API Rate Limiting — Design Fair and Useful Throttling

Overview

Rate limiting controls the number of API requests a client can make in a given time period. Done well it shields your backend from overload, shares resources fairly, and stops abuse — for the implementation, see the rate limiting recipe. Well-designed rate limits balance user experience with system protection.

This guide covers rate limiting algorithms, implementation strategies, and how to choose appropriate limits.

When to Use

• Your API is public-facing — bad actors can abuse it
• You have limited backend capacity and need to prevent overload
• You offer tiered service levels (free, pro, enterprise)
• You want to prevent cascading failures during traffic spikes
• You need to comply with partner API consumption agreements
• You’re formalizing the policy — the rate limiting policy template turns these decisions into a document your team signs

Skip rate limiting when the API has a single internal consumer: use timeouts, backpressure, and bounded queues instead — limits you control in code beat rejected requests. And if the real problem is cost per call rather than request volume, what you need is a usage quota, not a per-second limit.

Core Concepts

Concept	Description
Rate Limit	Maximum requests allowed per time window
Quota	Total allowance over a longer period (e.g., monthly API calls)
Throttling	Delaying or rejecting requests that exceed limits
Burst	Short-term allowance above steady-state rate
Window	Time period over which limits are enforced
Client Identity	How callers are identified (IP, API key, user ID, org ID)

Rate Limiting Algorithms

Token Bucket

Allows bursts up to bucket capacity while maintaining average rate:

import time
from threading import Lock

class TokenBucket:
    def __init__(self, capacity: int, refill_rate: float):
        self.capacity = capacity      # Maximum burst size
        self.tokens = capacity        # Current tokens available
        self.refill_rate = refill_rate  # Tokens added per second
        self.last_refill = time.time()
        self.lock = Lock()

    def allow_request(self, tokens: int = 1) -> bool:
        with self.lock:
            now = time.time()
            elapsed = now - self.last_refill
            self.tokens = min(
                self.capacity,
                self.tokens + elapsed * self.refill_rate
            )
            self.last_refill = now

            if self.tokens >= tokens:
                self.tokens -= tokens
                return True
            return False

# Example: 10 requests/second with burst of 20
bucket = TokenBucket(capacity=20, refill_rate=10)

Best for: APIs that need burst tolerance (e.g., user-facing APIs with sporadic traffic).

Leaky Bucket

Smooths out bursts into a steady flow rate:

import time
from collections import deque
from threading import Lock

class LeakyBucket:
    def __init__(self, capacity: int, leak_rate: float):
        self.capacity = capacity    # Maximum queue size
        self.leak_rate = leak_rate  # Requests processed per second
        self.queue = deque()
        self.last_leak = time.time()
        self.lock = Lock()

    def allow_request(self) -> bool:
        with self.lock:
            now = time.time()
            elapsed = now - self.last_leak
            # Remove processed requests from queue
            to_leak = int(elapsed * self.leak_rate)
            for _ in range(min(to_leak, len(self.queue))):
                self.queue.popleft()
            self.last_leak = now

            if len(self.queue) < self.capacity:
                self.queue.append(now)
                return True
            return False

Best for: Webhooks, processing pipelines, and situations requiring strict rate smoothing.

Sliding Window Log

Most accurate, most memory-hungry:

import time
from collections import deque
from threading import Lock

class SlidingWindowLog:
    def __init__(self, window_size: int, max_requests: int):
        self.window_size = window_size  # Seconds
        self.max_requests = max_requests
        self.requests = deque()
        self.lock = Lock()

    def allow_request(self) -> bool:
        with self.lock:
            now = time.time()
            cutoff = now - self.window_size

            # Remove requests outside the window
            while self.requests and self.requests[0] < cutoff:
                self.requests.popleft()

            if len(self.requests) < self.max_requests:
                self.requests.append(now)
                return True
            return False

Best for: Strict compliance requirements where exact enforcement matters.

Sliding Window Counter

Close approximation, much less memory:

import math
import time
from threading import Lock

class SlidingWindowCounter:
    def __init__(self, window_size: int, max_requests: int):
        self.window_size = window_size
        self.max_requests = max_requests
        self.current_window = int(time.time() // window_size)
        self.current_count = 0
        self.previous_count = 0
        self.lock = Lock()

    def allow_request(self) -> bool:
        with self.lock:
            now = int(time.time())
            window = now // self.window_size

            if window != self.current_window:
                self.previous_count = self.current_count
                self.current_count = 0
                self.current_window = window

            # Estimate requests in sliding window
            elapsed = now % self.window_size
            weight = 1 - (elapsed / self.window_size)
            estimated = (self.previous_count * weight) + self.current_count

            if estimated < self.max_requests:
                self.current_count += 1
                return True
            return False

Best for: High-traffic APIs where memory efficiency is important.

Choosing Rate Limits

Factors to Consider

Factor	Guideline
Endpoint cost	Expensive endpoints (ML, reports) get lower limits
User tier	Free: 100/hr, Pro: 10,000/hr, Enterprise: custom
Resource constraints	Limit based on backend capacity, not arbitrary numbers
Fairness	Per-user limits prevent one client from starving others
Business value	Protect revenue-generating endpoints most strictly

Example Tiered Limits

# Example: Tiered rate limits for a SaaS API
tiers:
  free:
    requests_per_minute: 60
    requests_per_hour: 1000
    requests_per_day: 10000
    burst: 10
  pro:
    requests_per_minute: 600
    requests_per_hour: 10000
    requests_per_day: 100000
    burst: 100
  enterprise:
    requests_per_minute: 6000
    requests_per_hour: 100000
    requests_per_day: 1000000
    burst: 1000

Implementation Strategies

Gateway-Level Rate Limiting

Enforce limits at the API gateway for centralized control:

# Example: NGINX rate limiting
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
map $http_x_api_key $api_key { default $binary_remote_addr; }
limit_req_zone $api_key zone=pro:10m rate=100r/s;

server {
    location /api/ {
        limit_req zone=api burst=20 nodelay;
        proxy_pass http://backend;
    }
}

Application-Level Rate Limiting

Fine-grained control within your application:

from fastapi import FastAPI, HTTPException, Request
from fastapi_limiter import FastAPILimiter
import redis.asyncio as redis
from fastapi import Depends
from fastapi_limiter.depends import RateLimiter

app = FastAPI()

@app.on_event("startup")
async def startup():
    app.state.redis = await redis.from_url("redis://localhost")
    await FastAPILimiter.init(app.state.redis)

@app.get("/api/data", dependencies=[Depends(RateLimiter(times=100, minutes=1))])
async def get_data(request: Request):
    # fastapi-limiter checks the request before the handler runs;
    # the key defaults to the client IP — supply a custom identifier
    # callback if you rate-limit by API key instead
    return {"data": "..."}

Distributed Rate Limiting

Share state across multiple instances:

# Redis-based distributed token bucket
import redis

class RedisTokenBucket:
    def __init__(self, redis_client: redis.Redis, key: str, capacity: int, refill_rate: float):
        self.redis = redis_client
        self.key = key
        self.capacity = capacity
        self.refill_rate = refill_rate

    def allow_request(self, tokens: int = 1) -> bool:
        lua_script = """
        local key = KEYS[1]
        local capacity = tonumber(ARGV[1])
        local refill_rate = tonumber(ARGV[2])
        local tokens_requested = tonumber(ARGV[3])
        local now = tonumber(ARGV[4])

        local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
        local current_tokens = tonumber(bucket[1]) or capacity
        local last_refill = tonumber(bucket[2]) or now

        local elapsed = now - last_refill
        local new_tokens = math.min(capacity, current_tokens + elapsed * refill_rate)

        if new_tokens >= tokens_requested then
            new_tokens = new_tokens - tokens_requested
            redis.call('HMSET', key, 'tokens', new_tokens, 'last_refill', now)
            redis.call('EXPIRE', key, 60)
            return 1
        else
            redis.call('HMSET', key, 'tokens', new_tokens, 'last_refill', now)
            redis.call('EXPIRE', key, 60)
            return 0
        end
        """
        return self.redis.eval(
            lua_script, 1, self.key,
            self.capacity, self.refill_rate, tokens, time.time()
        ) == 1

HTTP Response Headers

Communicate limits clearly to clients:

Header	Description	Example
X-RateLimit-Limit	Maximum requests allowed	100
X-RateLimit-Remaining	Requests remaining in window	42
X-RateLimit-Reset	Unix timestamp when limit resets	1704067200
Retry-After	Seconds to wait before retrying	60

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1704067200
Retry-After: 60

{
  "error": "Rate limit exceeded",
  "message": "You have exceeded 100 requests per minute. Retry after 60 seconds.",
  "retry_after": 60
}

Comparing the Algorithms

Algorithm	Exact?	Memory	Bursts	Best use
Fixed window	No — edge bursts	O(1)	Yes, uncontrolled	Internal APIs, rough protection
Token bucket	Approximate	O(1)	Yes, controlled	Public APIs — the common default
Leaky bucket	Exact outflow	O(n) queue	No — smooths to steady rate	Webhooks, processing pipelines
Sliding log	Exact	O(n) per client	No	Compliance-critical enforcement
Sliding counter	~99% accurate	O(1)	No	High-traffic APIs

The real trade-off is precision versus memory. A sliding log stores one timestamp per request per client — fine for 1,000 clients at 100 requests, brutal at a million. Sliding window counter gets within a few percent of the log’s accuracy for O(1) memory by weighting the previous window’s count. Token and leaky buckets answer different questions: token bucket asks “may this request in?” while leaky bucket asks “when may it go out?” Pick the second only when a steady outflow matters more than burst tolerance.

When the Rate-Limit Store Fails

Distributed limiters have a failure mode nobody sees until it happens: Redis goes down and every allow_request either errors or silently returns false. Decide the behavior explicitly:

• Fail-open: allow requests when the store is unreachable. Keeps the API up during an outage — and lets a real attack through during exactly that window.
• Fail-closed: reject (or return a safe 429) when the store is unreachable. Protects the backend; turns a Redis blip into a customer-facing outage.
• Local fallback: keep a small in-process token bucket as the last line. Limits degrade to per-instance values — looser, but bounded and predictable.

Most teams pick fail-open with a local fallback and alert hard on 429 anomalies. Whatever you choose, put it in the policy document — “what the limiter does when its store is down” is a product decision, not an implementation detail.

Tuning Client Behavior

A rate limit is a contract, and the headers are the readable part. Three rules that keep clients honest: set Retry-After from the actual window reset, not a guess; send X-RateLimit-Remaining on success responses too, not only on 429s, so well-behaved clients can self-throttle before they hit the wall; and document the exact reset semantics (fixed boundary versus rolling). For clients, the correct reaction to a 429 is exponential backoff with jitter — retrying at exactly Retry-After in lockstep across thousands of clients recreates the spike you just rejected.

Testing and Monitoring Limits

Limits that are never tested are usually broken. Three cheap checks belong in CI: fire requests past the limit and assert the first 429 arrives exactly at limit + 1; verify the response carries Retry-After and the JSON body your docs promise; and run the whole thing with the store clock skewed — window-edge bugs only show up when two requests straddle a boundary. In production, alert on the 429 rate, not just the absolute count: a sudden jump means either an integration error or someone probing. A slow, steady climb means your limits are too tight for real usage — or that a misbehaving client needs a conversation, not a bigger quota.

Sizing a Limit from Capacity

Work backwards from what the backend can serve, not forwards from what sounds reasonable. If the service sustains 2,000 RPS and POST /orders costs three times a cheap read, the write budget is roughly 2,000 / (1 + 2·write_fraction). Start conservative — a limit that is too tight for two weeks is fixable; a limit that is too loose during a real incident is not. Then instrument: publish the numbers to the client-facing docs, because a limit nobody knows about gets blamed as a bug.

Rate Limiting in the Bigger Picture

Rate limiting is one layer, not the whole defense. Quotas bound total consumption over a billing period — a client inside its per-second limit can still burn a million calls a month. Concurrency limits cap in-flight requests and protect thread pools when requests get slow; a rate limit alone does nothing when every request takes 30 seconds because they all “fit” the window. Cost-based throttling treats an expensive report as a dozen cheap reads rather than one request. And a WAF or bot layer in front handles what rate limits cannot: traffic you do not want at all, not just too much of it.

Get the layering order right and each piece stays simple. The typical stack: WAF drops obviously hostile traffic, the gateway enforces coarse per-key limits, per-service limiters protect hot paths inside, and quotas handle the business dimension. Identity matters as much as numbers — an API-key limit fails when one customer issues ten keys, so map limits to the identity that actually pays or causes load (org ID, not key).

Communicating Limits to Clients

Publishing numbers is half the job; publishing the semantics is the rest. Clients need to know the window model (fixed versus rolling), whether limits are per key, per org, or per endpoint, what the burst allowance is, and what the 429 body looks like. A docs page with only “100 req/min” produces a support ticket the first time a client hits the invisible hourly quota.

Make the limits visible where they hurt least: in the developer dashboard, in the onboarding flow, and inside the SDK if you ship one — a client that handles Retry-After automatically turns a 429 from an error into a speed bump. When you change limits, treat it like a breaking change: advance notice, a deprecation window with headers announcing the new ceiling, and an explicit announcement channel. Surprising a paying customer with a tighter limit costs more trust than the overload it was meant to prevent.

One pattern that pays for itself quickly is a dedicated “limits” page that shows each tier’s numbers alongside the client’s current usage — the same numbers the headers carry, rendered as progress bars. When a customer can watch themselves approach the ceiling, the 429 stops being a surprise and starts being a signal they acted on in advance. Pair it with a self-service upgrade path and the limit itself becomes a growth funnel: the client who keeps hitting the free tier’s wall is your warmest upgrade lead. Abuse protection and sales rarely share a document; rate limiting is where they meet.

Finally, rehearse the failure before production does it for you. Point a load generator at a staging gateway with half your planned limits and watch what the 429 rate, the client retries, and the backend CPU do — the ten minutes of data from that drill settles most arguments about which numbers to ship. Keep the drill repeatable — a five-line k6 script and a staging flag are enough — and rerun it whenever the backend topology changes. A limit tuned for three app instances surprises you badly the day autoscaler brings up fifteen: the same per-instance bucket math now admits five times the traffic. Revisiting the numbers after every capacity change is cheap insurance against discovering your limits wrong during the next launch spike — the worst possible moment to find out the contract you published no longer matches the infrastructure you run. Write the review date into the policy itself, and treat a missed review like a missed deployment gate — because that is exactly what it is in practice too — every single time.

What Works

• Return informative error messages. Tell clients exactly what limit they hit and when they can retry.
• Use different limits per endpoint. Search endpoints may tolerate higher limits than write endpoints.
• Implement exponential backoff on clients. 429 responses should trigger backoff, not immediate retries.
• Monitor rate limit hits. Sudden spikes in 429s may indicate attacks or integration issues.
• Allow grace periods for new clients. Start with generous limits and tighten based on usage patterns.
• Document your limits. Publish them in the API docs.

Common Mistakes

• Using IP addresses as the only identifier. NAT and mobile networks share IPs; use API keys or user IDs.
• Noisy neighbor problems. One heavy user shouldn’t impact others; enforce per-client limits.
• Ignoring burst traffic. Legitimate users may burst during page loads; allow short bursts.
• Inconsistent limits across services. Standardize limits by tier and endpoint type.
• Forgetting to handle edge cases. What happens when the rate limit store (Redis) is down?

Variants

• Concurrency limiting: cap simultaneous in-flight requests instead of rate over time.
• Adaptive rate limiting: Dynamically adjust limits based on backend health (lower limits when overloaded).
• Geographic rate limiting: different limits per client location or regulatory need.
• Cost-based throttling: Limit expensive operations (ML inference, report generation) more strictly.

Further Reading

• RFC 9110 — HTTP Semantics, 429 Too Many Requests — the status code and Retry-After semantics
• IETF draft — RateLimit Fields for HTTP — the standardizing header family
• Redis — Rate limiter pattern — fixed-window Lua implementation on Redis
• NGINX — ngx_http_limit_req_module — the module behind the gateway example