Throttling Pattern

Overview

The Throttling Pattern controls the rate at which a system processes requests or consumes resources to prevent overload and ensure fair resource allocation. Instead of accepting all incoming requests immediately, the system limits the rate based on capacity, user tiers, or resource availability.

Throttling prevents cascading failures by ensuring downstream services and shared resources are not overwhelmed. It is commonly used in APIs, message consumers, database connections, and third-party integrations where unbounded throughput could cause service degradation or cost explosion.

When to Use

• For alternatives, see Content Delivery Network (CDN) Pattern.

• Protecting downstream services from traffic spikes

• Enforcing API rate limits for consumers

• Controlling database connection pool exhaustion

• Managing costs with metered third-party APIs

• Ensuring fair resource allocation in multi-tenant systems

• Preventing DDoS or accidental abuse

When to Avoid

• Internal services within the same trust boundary with predictable load
• Systems where any request rejection violates business requirements
• When the bottleneck is not request rate but data size or complexity
• Latency-sensitive paths where throttling adds unacceptable delay

Solution

Python (Token Bucket)

import time
import threading
from dataclasses import dataclass

@dataclass
class TokenBucket:
    capacity: int
    refill_rate: float
    tokens: float = 0
    last_refill: float = 0
    lock: threading.Lock = None

    def __post_init__(self):
        self.tokens = self.capacity
        self.last_refill = time.time()
        self.lock = threading.Lock()

    def acquire(self, tokens: int = 1) -> bool:
        with self.lock:
            now = time.time()
            elapsed = now - self.last_refill
            self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
            self.last_refill = now

            if self.tokens >= tokens:
                self.tokens -= tokens
                return True
            return False

class ThrottledAPI:
    def __init__(self):
        self.bucket = TokenBucket(capacity=10, refill_rate=2)

    def call(self, endpoint: str, data: dict) -> dict:
        if not self.bucket.acquire():
            raise RateLimitExceeded("Rate limit exceeded. Try again later.")

        # Process request
        return {"status": "success", "endpoint": endpoint}

class RateLimitExceeded(Exception):
    pass

Java (Guava RateLimiter)

import com.google.common.util.concurrent.RateLimiter;
import org.springframework.stereotype.Service;

@Service
public class ThrottledService {
    private final RateLimiter limiter = RateLimiter.create(10.0); // 10 permits/second

    public String processRequest(String request) {
        limiter.acquire(); // Blocks until permit available
        return "Processed: " + request;
    }
}

JavaScript (Sliding Window Log)

class SlidingWindowThrottle {
    constructor(windowMs, maxRequests) {
        this.windowMs = windowMs;
        this.maxRequests = maxRequests;
        this.requests = new Map();
    }

    isAllowed(clientId) {
        const now = Date.now();
        const windowStart = now - this.windowMs;

        if (!this.requests.has(clientId)) {
            this.requests.set(clientId, []);
        }

        const clientRequests = this.requests.get(clientId);
        const recent = clientRequests.filter(t => t > windowStart);

        if (recent.length < this.maxRequests) {
            recent.push(now);
            this.requests.set(clientId, recent);
            return true;
        }

        this.requests.set(clientId, recent);
        return false;
    }
}

Explanation

Throttling algorithms balance fairness and efficiency:

• Token bucket: Tokens are added at a fixed rate. Requests consume tokens. Allows short bursts while maintaining long-term average rate.
• Leaky bucket: Requests enter a fixed-size queue and leak out at a constant rate. Smooths traffic but drops overflow.
• Fixed window: Count requests in time windows. Simple but allows burst at window boundaries.
• Sliding window: More accurate by tracking exact timestamps within a rolling window.

Variants

Variant	Behavior	Best For
Token bucket	Bursts allowed up to capacity	APIs needing burst tolerance
Leaky bucket	Constant outflow rate	Smoothing traffic to downstream
Fixed window	Reset counter per interval	Simple implementations
Sliding window	Rolling time window	Accurate per-client rate limits

What Works

• Return 429 Too Many Requests with Retry-After header for HTTP APIs
• Differentiate between user tiers with different limits
• Monitor rejection rates as an early warning signal
• Implement backoff for clients that are throttled
• Consider distributed rate limiting for multi-instance deployments

Common Mistakes

• Throttling without communicating limits to clients
• Using same limits for all users regardless of tier
• Not handling clock skew in distributed systems
• Forgetting to clean up expired entries in window-based algorithms

Real-World Examples

GitHub API

GitHub enforces rate limits per authenticated user (5000 requests/hour) and per IP (60 requests/hour). Exceeding limits returns 403 with X-RateLimit-Reset header.

AWS API Gateway

API Gateway supports throttling at account, stage, and method levels using token bucket algorithms, with burst capacity for traffic spikes.

Troubleshooting

• Pattern does not fit the problem: re-evaluate the forces (performance, scalability, team size, coupling). A pattern is only appropriate when its trade-offs match your constraints.
• Too many abstractions: if adding a pattern increases complexity without a clear benefit, simplify. Not every module needs a factory, decorator, or strategy.
• Tight coupling after refactoring: check that interfaces are stable and dependencies point inward.
• Tests break when the design changes: favor stable contracts over internal structure.
• Performance regression from indirection: measure before and after. Layers, decorators, and adapters can add latency; cache or inline hot paths if needed.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply throttling pattern when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Advanced Topics

Scenario: Throttling for Geolocation API

// Throttling pattern: max 10 requests per second
class Throttle {
  private requests: number[] = [];
  constructor(private maxRequests: number, private windowMs: number) {}

  canProceed(): boolean {
    const now = Date.now();
    this.requests = this.requests.filter(t => now - t < this.windowMs);
    if (this.requests.length < this.maxRequests) {
      this.requests.push(now);
      return true;
    }
    return false;
  }
  timeUntilNextSlot(): number {
    if (this.requests.length < this.maxRequests) return 0;
    const oldest = this.requests[0];
    return this.windowMs - (Date.now() - oldest);
  }
}

// Usage: Google Maps API (limit 10 req/s)
const throttle = new Throttle(10, 1000);

async function geocode(address: string): Promise<LatLng> {
  if (!throttle.canProceed()) {
    const wait = throttle.timeUntilNextSlot();
    await new Promise(resolve => setTimeout(resolve, wait));
  }
  const response = await fetch(`https://maps.googleapis.com/maps/api/geocode/json?address=${address}`);
  return response.json();
}

// Comparison: Throttle vs Rate Limit vs Debounce
  | Pattern | Purpose | Example |
  |---------|---------|---------|
  | Throttle | Max N requests per window | 10 req/s |
  | Rate Limit | Reject if exceeded | 429 Too Many Requests |
  | Debounce | Wait for input to stop | Search autocomplete |
  | Token Bucket | Tokens refill over time | Burst + sustained |
  | Leaky Bucket | Queue with constant output | Smooth bursts |

Lessons:

• Throttle limits request rate: does not reject, waits
• Rate limit rejects: 429 with Retry-After header
• Debounce groups calls: waits for inactivity
• Token bucket allows burst: useful for APIs with quotas
• Measure actual throughput: do not assume the limit is exact

### How do I choose between throttle and rate limit?

Use throttle when the client should wait (e.g: calling external API with limit). Use rate limit when the client should be rejected (e.g: protecting your API from abuse). Throttle is cooperative: the client self-limits. Rate limit is imposed: the server rejects. For public APIs, use rate limit (429 + Retry-After). For internal integrations, throttle is sufficient.














End of document. Review and update quarterly.

## Common Production Pitfalls

- Applying the pattern where no abstraction is needed, adding accidental complexity.
- Letting the pattern leak into unrelated modules and blur ownership boundaries.
- Over-engineering the first implementation instead of starting simple and measuring pain.
- Skipping contract tests, so refactors silently break consumers.
- Ignoring failure modes that the pattern does not cover.
- Using the pattern as a default instead of choosing the right tool for the current scale.
- Forgetting to document when to stop using the pattern and what replaces it.
- Missing observability around the pattern's performance and error propagation.