Serverless Throttling Pattern

Overview

Serverless functions scale automatically, but downstream services (databases, APIs) often cannot handle the burst traffic that Lambda concurrency produces. The throttling pattern controls the rate at which Lambda functions process events, protecting downstream systems from overload.

Three main approaches: SQS with controlled batch processing, reserved concurrency limits, and token bucket rate limiting. Each controls throughput at a different level: queue-based throttling smooths traffic over time, reserved concurrency caps parallel executions, and token buckets enforce a precise request rate.

When to Use

• Lambda bursts overwhelm downstream databases or APIs with connection limits
• You need to maintain a specific request rate to a third-party API with rate limits
• Traffic spikes cause downstream timeouts or connection pool exhaustion
• You want to process messages at a controlled pace rather than as fast as possible
• Multiple Lambda functions compete for a shared downstream resource

Solution

Strategy 1: SQS with Controlled Batch Processing (Python)

import boto3
import json
import time

sqs = boto3.client('sqs')
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/order-queue"

def handler(event, context):
    # Process messages in small batches with deliberate pacing
    batch_size = 5
    processed = 0

    for record in event["Records"]:
        message = json.loads(record["body"])

        try:
            process_order(message)
            processed += 1
        except Exception as e:
            # Return message to queue for retry
            raise e

    return {"processed": processed}

def process_order(order: dict):
    # Simulate downstream API call with rate limit awareness
    api_client.post("/orders", json=order)

CDK: SQS with Visibility Timeout and Batch Size

from aws_cdk import (
    Stack,
    aws_sqs as sqs,
    aws_lambda as lambda_,
    aws_lambda_event_sources as events,
    Duration,
)
from constructs import Construct

class ThrottledConsumerStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs):
        super().__init__(scope, construct_id, **kwargs)

        # Queue with long visibility timeout to prevent concurrent processing
        queue = sqs.Queue(self, "ThrottledQueue",
            visibility_timeout=Duration.seconds(300),
            receive_message_wait_time=Duration.seconds(20),  # Long polling
        )

        # Dead-letter queue for failed messages
        dlq = sqs.Queue(self, "DeadLetterQueue",
            retention_period=Duration.days(14),
        )

        # Consumer Lambda with limited concurrency
        consumer = lambda_.Function(self, "ThrottledConsumer",
            runtime=lambda_.Runtime.PYTHON_3_12,
            handler="handler.handler",
            code=lambda_.Code.from_asset("lambda/consumer"),
            timeout=Duration.seconds(60),
            reserved_concurrent_executions=10,  # Cap parallelism
        )

        # Small batch size to control throughput
        consumer.add_event_source(events.SqsEventSource(
            queue,
            batch_size=5,
            max_batching_window=Duration.seconds(30),
            report_batch_item_failures=True,
        ))

        # DLQ subscription
        queue.add_to_dead_letter_queue(
            max_receive_count=5,
            dead_letter_queue=dlq,
        )

Strategy 2: Token Bucket Rate Limiter (TypeScript)

import { DynamoDBClient, UpdateItemCommand, GetItemCommand } from '@aws-sdk/client-dynamodb';
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb';

const ddb = new DynamoDBClient({ region: 'us-east-1' });

class TokenBucket {
  private tableName = 'rate_limits';
  private capacity: number;
  private refillRate: number;  // tokens per second

  constructor(capacity: number, refillRate: number) {
    this.capacity = capacity;
    this.refillRate = refillRate;
  }

  async allowRequest(bucketKey: string, tokens = 1): Promise<boolean> {
    const now = Date.now();

    // Get current bucket state
    const getResponse = await ddb.send(new GetItemCommand({
      TableName: this.tableName,
      Key: marshall({ bucketKey }),
    }));

    let tokensAvailable = this.capacity;
    let lastRefill = now;

    if (getResponse.Item) {
      const bucket = unmarshall(getResponse.Item);
      tokensAvailable = parseFloat(bucket.tokens);
      lastRefill = parseInt(bucket.lastRefill);

      // Refill tokens based on elapsed time
      const elapsedSeconds = (now - lastRefill) / 1000;
      const refilled = elapsedSeconds * this.refillRate;
      tokensAvailable = Math.min(this.capacity, tokensAvailable + refilled);
    }

    if (tokensAvailable < tokens) {
      return false;  // Rate limit exceeded
    }

    // Consume tokens
    tokensAvailable -= tokens;

    await ddb.send(new UpdateItemCommand({
      TableName: this.tableName,
      Key: marshall({ bucketKey }),
      UpdateExpression: 'SET tokens = :t, lastRefill = :r',
      ExpressionAttributeValues: marshall({
        ':t': tokensAvailable.toFixed(2),
        ':r': now.toString(),
      }),
    }));

    return true;
  }
}

// Usage in Lambda handler
const rateLimiter = new TokenBucket(100, 10);  // 100 tokens, 10/sec refill

export const handler = async (event: any): Promise<{ statusCode: number }> => {
  const apiKey = event.requestContext.identity.apiId;

  const allowed = await rateLimiter.allowRequest(apiKey);
  if (!allowed) {
    return {
      statusCode: 429,
      body: JSON.stringify({ error: 'Rate limit exceeded' }),
    };
  }

  // Process request
  return { statusCode: 200 };
};

Strategy 3: Reserved Concurrency + SQS (Python)

import boto3
import json
import os

sqs = boto3.client('sqs')
QUEUE_URL = os.environ["QUEUE_URL"]

def handler(event, context):
    failed_records = []

    for record in event["Records"]:
        try:
            message = json.loads(record["body"])
            process_with_rate_limit(message)
        except Exception as e:
            # Report individual failure for partial batch retry
            failed_records.append({
                "itemIdentifier": record["messageId"]
            })

    if failed_records:
        return {
            "batchItemFailures": failed_records
        }

    return {"batchItemFailures": []}

def process_with_rate_limit(message: dict):
    # Check downstream capacity before processing
    if not check_downstream_capacity():
        raise Exception("Downstream capacity exceeded")

    # Process with timeout
    api_client.post("/process", json=message, timeout=10)

def check_downstream_capacity() -> bool:
    # Query downstream health endpoint
    response = api_client.get("/health")
    return response.status_code == 200 and response.json()["capacity"] > 0

Explanation

Each throttling strategy controls throughput at a different level:

• SQS batch control — the queue buffers incoming traffic. Lambda processes small batches (5-10 messages) with a batching window (30s). This smooths bursts: messages wait in the queue instead of hitting downstream all at once. The visibility timeout prevents duplicate processing during long-running invocations.

• Reserved concurrency — AWS caps the number of concurrent Lambda executions. Setting reserved_concurrent_executions=10 means at most 10 instances run simultaneously. Excess invocations queue up. This directly limits parallel load on downstream systems.

• Token bucket — a DynamoDB-backed counter tracks available tokens. Each request consumes a token. Tokens refill at a fixed rate. If no tokens are available, the request is rejected with 429. This enforces a precise request rate regardless of Lambda concurrency.

Variants

Approach	Control Level	Best For
SQS batch + batching window	Queue-level	Smoothing traffic bursts
Reserved concurrency	Execution-level	Capping parallel invocations
Token bucket (DynamoDB)	Request-level	Precise rate per API key
Leaky bucket	Request-level	Strict rate with no bursts
Circuit breaker	Downstream-level	Protecting failing services
Semaphore (in-function)	Instance-level	Limiting concurrent async calls

Best Practices

• For a deeper guide, see Serverless Fanout Pattern.

• Combine SQS with reserved concurrency — SQS buffers traffic; reserved concurrency caps parallelism. Together they provide both smoothing and hard limits.

• Use partial batch failure reporting — when one message in a batch fails, return only that message ID in batchItemFailures. SQS re-delivers only the failed message, not the entire batch.

• Set visibility timeout to 6x Lambda timeout — prevents redelivery while a long-running invocation is still processing.

• Monitor Throttles metric — if Lambda throttles frequently, increase reserved concurrency or scale downstream capacity. Throttles mean lost invocations.

• Use long polling — set receive_message_wait_time to 20 seconds. This reduces empty receives and lowers SQS API costs.

Common Mistakes

• No reserved concurrency — without a cap, Lambda scales to thousands of concurrent executions, overwhelming downstream databases. Always set reserved concurrency for functions that call downstream services.
• Batch size too large — processing 100 messages at once increases the chance of partial failures and long-running invocations. Use 5-10 for most workloads.
• No visibility timeout — without a visibility timeout, SQS redelivers messages while Lambda is still processing them, causing duplicates. Set it to 6x the Lambda timeout.
• Ignoring downstream capacity — throttling at the Lambda level does not help if the downstream service has its own connection limits. Monitor downstream health and adjust concurrency accordingly.
• Not handling 429 from downstream — when a downstream API returns 429, the Lambda should retry with backoff or fail the message for later reprocessing, not crash.

Troubleshooting

• Cold start latency is high: increase provisioned concurrency, reduce package size, and avoid initializing heavy clients per invocation.
• Function times out: check downstream dependencies, memory allocation, and retry logic. Increase timeout only after optimizing the code.
• State lost between invocations: serverless functions are stateless. Persist state in a database, cache, or durable queue.
• Deployment package too large: exclude dev dependencies and unused assets.
• Event ordering issues: many event sources are at-least-once and unordered. Design for idempotency and explicit sequencing.

Key Takeaways

• Apply serverless throttling pattern when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Common Production Pitfalls

• Applying the pattern where no abstraction is needed, adding accidental complexity.
• Letting the pattern leak into unrelated modules and blur ownership boundaries.
• Over-engineering the first implementation instead of starting simple and measuring pain.
• Skipping contract tests, so refactors silently break consumers.
• Ignoring failure modes that the pattern does not cover.
• Using the pattern as a default instead of choosing the right tool for the current scale.
• Forgetting to document when to stop using the pattern and what replaces it.
• Missing observability around the pattern’s performance and error propagation.