API Gateway Design: Resilience, Routing, and Security

Overview

Every microservices architecture eventually needs a front door. The API gateway is that door — and if you design it poorly, it becomes a single point of failure, a performance bottleneck, or a security gap. A well-designed gateway handles routing, authentication, rate limiting, circuit breaking, caching, and observability so that individual services can focus on business logic.

This guide walks through what happens to a request inside the gateway, the responsibilities worth centralizing there (and the ones that aren’t), working configurations for Envoy, NGINX, Kong, and middleware code, and the deployment patterns that keep the gateway itself from becoming the thing that takes your platform down.

A Request Through the Gateway

Before picking technologies, it helps to see the pipeline. A typical request moves through the gateway in a fixed order — each stage either rejects it early or decorates it and passes it on:

Two properties of that pipeline matter when you design for failure. First, order is deliberate: authentication runs before rate limiting so limits apply per identity, not per IP that might be a shared NAT. Second, the cache sits before the circuit breaker so a dead upstream can still serve stale-but-valid responses instead of cascading 503s.

Core Responsibilities

Concern	Implementation Approach	Key Technology
Request Routing	Path-, header-, or host-based routing to upstream services	NGINX, Kong, Envoy, AWS API Gateway
Rate Limiting	Token bucket or leaky bucket per client, IP, or API key	Redis + Lua, Envoy, Kong
Authentication	Validate JWT, API keys, or OAuth2 tokens at the edge	Keycloak, Auth0, custom middleware
Circuit Breaker	Track failure rates and fail fast when upstreams are unhealthy	Resilience4j, Envoy, Polly
Load Balancing	Round-robin, least connections, or consistent hashing	Envoy, NGINX, service mesh
Response Caching	Cache GET responses per key or route	Kong proxy-cache, NGINX proxy_cache, Redis
Observability	Distributed tracing, metrics, and access logs	OpenTelemetry, Prometheus, Fluentd
TLS Termination	Offload TLS at the gateway to reduce CPU load on services	NGINX, HAProxy, cloud load balancers
Request/Response Transformation	Header injection, payload rewriting, protocol translation	Kong plugins, Envoy filters, middleware

The dividing line worth respecting: the gateway owns concerns that are identical for every service (auth, limits, routing, tracing). Anything that requires knowing the business — pricing rules, inventory checks, validation tied to a specific domain — belongs in the service. Teams that blur this line end up with a “distributed monolith” where a deploy of business logic requires touching gateway config.

Gateway vs Load Balancer vs BFF vs Service Mesh

These four get confused constantly, and choosing the wrong one is expensive:

Component	What it does	When you need it
Load balancer	Distributes traffic across instances of the same service	Always — usually beneath the gateway itself
API gateway	Routes to different services; owns auth, limits, transformation	Multiple services, external or partner-facing APIs
BFF (Backend for Frontend)	A per-client-type aggregation layer (web vs mobile payloads)	Clients with materially different data needs
Service mesh	East-west traffic between services: mTLS, retries, telemetry	Internal service-to-service communication at scale

A gateway often uses a load balancer internally per upstream — they’re complementary, not alternatives. A common production layout is gateway for north-south traffic plus a mesh for east-west; the mistake is bolting an external gateway onto service-to-service calls, where it adds a hop and a failure domain for zero benefit.

Routing

Routing decisions should be stateless and deterministic so any gateway instance can handle any request. Session affinity is a smell — if upstreams need sticky sessions, prefer consistent hashing over IP-based stickiness, or fix the service to externalize state.

# Envoy configuration snippet
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/api/users" }
                          route: { cluster: user_service, prefix_rewrite: "/" }
                        - match: { prefix: "/api/orders" }
                          route: { cluster: order_service, prefix_rewrite: "/" }
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: user_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: user_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address: { address: user-service, port_value: 8080 }

The same routes in NGINX look simpler when you don’t need Envoy’s filter ecosystem:

upstream user_service {
    server user-service:3000;
}

server {
    listen 80;

    location /api/users/ {
        proxy_pass http://user_service;
        proxy_set_header Host $host;
        proxy_set_header X-Request-ID $request_id;
    }

    location / {
        return 404;  # unmatched routes die at the edge, not upstream
    }
}

Authentication at the Edge

The gateway validates who the caller is and forwards the result as headers — services shouldn’t re-verify signatures or re-parse tokens. What it shouldn’t do is decide what the caller can do: fine-grained authorization stays in services that own the business rules.

// gateway/middleware/auth.ts — Validate JWT before reaching backend
import jwt from 'jsonwebtoken';

interface AuthConfig {
  jwtSecret: string;
  publicPaths: string[];  // paths that skip auth
}

function authMiddleware(config: AuthConfig) {
  return (req: Request, res: Response, next: NextFunction) => {
    if (config.publicPaths.some(path => req.path.startsWith(path))) {
      return next();
    }

    const token = req.headers.authorization?.replace('Bearer ', '');
    if (!token) {
      return res.status(401).json({ error: 'Missing authorization header' });
    }

    try {
      const payload = jwt.verify(token, config.jwtSecret) as JwtPayload;

      // Inject identity for upstream services — they trust these headers
      req.headers['x-user-id'] = payload.sub;
      req.headers['x-user-roles'] = payload.roles.join(',');
      req.headers['x-user-scopes'] = payload.scopes.join(',');

      next();
    } catch (error) {
      if (error instanceof jwt.TokenExpiredError) {
        return res.status(401).json({ error: 'Token expired' });
      }
      return res.status(401).json({ error: 'Invalid token' });
    }
  };
}

Two hardening details worth copying: reject before doing any work (the public-path check and missing-token check happen before jwt.verify), and validate against a rotated public key fetched from your identity provider’s JWKS endpoint rather than a shared secret when you can — symmetric secrets in gateway config tend to outlive their rotation schedule.

For machine-to-machine clients, API keys are simpler: look the key up, check expiry, apply its rate limit tier, and inject x-api-key-id plus x-tenant-id for upstream use. Keys belong in a store you can rotate, not in config files — for the broader auth surface see the API security checklist.

Rate Limiting

Rate limiting must be distributed in a multi-instance gateway deployment. In-memory counters per instance let a client round-robin across replicas and multiply its allowed quota — the single most common rate limiting bug in production. Keep counters in Redis (or your gateway’s shared store) and enforce per identity, not just per IP.

# Token bucket rate limiter backed by Redis
import time
import redis

class TokenBucketRateLimiter:
    def __init__(self, redis_client, key_prefix, capacity, refill_rate):
        self.redis = redis_client
        self.key_prefix = key_prefix
        self.capacity = capacity
        self.refill_rate = refill_rate

    def is_allowed(self, client_id):
        key = f"{self.key_prefix}:{client_id}"
        pipeline = self.redis.pipeline()
        now = time.time()
        pipeline.hmget(key, ["tokens", "last_refill"])
        result = pipeline.execute()
        tokens = float(result[0][0] or self.capacity)
        last_refill = float(result[0][1] or now)

        elapsed = now - last_refill
        tokens = min(self.capacity, tokens + elapsed * self.refill_rate)

        if tokens >= 1:
            tokens -= 1
            self.redis.hmset(key, {"tokens": tokens, "last_refill": now})
            self.redis.expire(key, 60)
            return True
        else:
            self.redis.hmset(key, {"tokens": tokens, "last_refill": last_refill})
            return False

Tiers usually map to commercial plans — for example free 100 req/min, pro 1.000 req/min, enterprise 10.000 req/min — and the limiter key should combine identity and tier, not just a client ID. When you reject, return 429 with Retry-After and X-RateLimit-Remaining on success responses; clients that can see their budget behave better. For dedicated strategies (leaky bucket, sliding window, quota vs spike limits), see the API rate limiting guide.

Resilience: Circuit Breakers and Timeouts

Circuit breakers prevent cascading failures. When an upstream struggles, the gateway should fail fast rather than queue requests that will all time out — this protects both the struggling service and the gateway’s own thread pools.

// Resilience4j circuit breaker configuration
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)
    .slowCallRateThreshold(80)
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .permittedNumberOfCallsInHalfOpenState(10)
    .slidingWindowSize(100)
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .build();

CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);
CircuitBreaker userServiceCB = registry.circuitBreaker("userService");

Supplier<String> decorated = CircuitBreaker
    .decorateSupplier(userServiceCB, () -> userClient.getUser(userId));

The half-open state is the part people get wrong: after the cooldown (waitDurationInOpenState), only a few probe requests go through — not full traffic — so a flapping upstream can’t oscillate the breaker. Pair breakers with explicit upstream timeouts (per service, not a global default, because a search endpoint and a health check have different latency profiles), and retry only idempotent requests with backoff — retrying a POST /payments is how you double-charge customers.

Caching and Transformation

Caching GET responses at the edge cuts upstream load and improves latency, but do it per key or route with explicit TTLs — and never cache responses carrying per-user data without the identity in the cache key:

// gateway/middleware/cache.ts — Cache GET responses with per-user keys
function cacheMiddleware(cache: ResponseCache, defaultTtl = 60) {
  return async (req: Request, res: Response, next: NextFunction) => {
    if (req.method !== 'GET') return next();

    const cacheKey = `${req.headers['x-user-id']}:${req.path}:${JSON.stringify(req.query)}`;
    const cached = await cache.get(cacheKey);

    if (cached) {
      res.setHeader('X-Cache', 'HIT');
      return res.status(cached.status).send(cached.body);
    }

    const originalSend = res.send;
    res.send = function (data: any) {
      if (res.statusCode === 200) {
        const ttl = parseInt(res.getHeader('Cache-Control')?.toString().match(/max-age=(\d+)/)?.[1] || '') || defaultTtl;
        cache.set(cacheKey, data.toString(), res.statusCode, ttl);
        res.setHeader('X-Cache', 'MISS');
      }
      return originalSend.call(this, data);
    };

    next();
  };
}

Transformation is the other edge concern worth centralizing — request IDs injected for tracing, /api/v2/users rewritten to an x-api-version header plus a versionless path, response envelopes standardized. Protocol translation (REST in, gRPC out) follows the same principle: map method+path pairs to gRPC calls at the edge so internal services stay protocol-agnostic. Keep the mapping table explicit in config — implicit translation rules become archaeology within a year.

Observability at the Edge

The gateway is the only point that sees every request, which makes it the cheapest instrumentation spot in the whole system. Three things to collect there:

• RED metrics per route and per upstream: request rate, error rate, and duration histograms. A spike in p99 latency on /api/orders tells you where to look before any service-level dashboard does.
• Request IDs end to end: generate X-Request-ID at the edge when absent, echo it in the response, and forward it upstream so a support ticket maps to one ID across every log line.
• Trace context propagation: honor traceparent/tracestate headers (W3C Trace Context) or generate a root span — the gateway span is the natural parent for everything downstream.

Access logs deserve their own mention. Emit structured JSON — timestamp, route, upstream, status, latency, bytes, request ID, client identity — and keep auth tokens and query strings carrying credentials out of them. A single access_token leaked into a shared log bucket is a security incident, not a log line.

Two failure modes to watch: unbounded cardinality (per-user label values will kill your metrics backend — keep labels to route, upstream, and status class), and sampling too aggressively at the edge, which hides slow upstreams exactly when they matter.

Sizing and Capacity Planning

Gateways are usually CPU-bound — TLS handshakes, JWT verification, payload rewriting — rather than memory-bound, so size them like proxies, not application servers. A rough method:

1. Measure peak RPS at the edge, not per service; the gateway sees all of it, retries included.
2. Benchmark one replica’s sustainable RPS with the real plugin set enabled. Plugin overhead adds up fast: a JWT check is cheap, a per-request Lua transform isn’t.
3. Run at 60–70% of that benchmark during peak and keep N+1 capacity, so losing one replica doesn’t start a cascade.
4. Track connection state separately: each replica holds upstream connections, so a retry storm can exhaust file descriptors before it exhausts CPU.

Long-lived connections change this math entirely, which is why they get their own treatment.

WebSockets and Server-Sent Events

Persistent connections break several gateway assumptions. Timeouts tuned for request/response traffic will sever WebSocket sessions mid-flight — most gateways need a read_timeout of minutes to hours on WS routes, plus explicit Upgrade/Connection header handling so the upgrade handshake survives. Load balancing shifts too: round-robin spreads upgrade requests evenly, but each socket then lives on a single upstream instance, so draining that instance requires connection migration support or clients that reconnect cleanly. Rate limiting still applies to the upgrade handshake, not to each message on the socket — enforcing per-message limits at the gateway adds latency without value; that logic belongs in the application.

Versioning and Deprecation

API versioning at the gateway is mostly a routing problem with a policy layer on top. The common models:

• Path-based (/api/v1/users, /api/v2/users): the most explicit and the easiest to route — each version prefix maps to a different upstream or upstream version. Also the easiest to cache and to document.
• Header-based (Accept-Version: 2 or a media type like application/vnd.acme.v2+json): keeps URLs clean, but makes every routing rule depend on header inspection and complicates caching and client debugging.
• Query-based (?version=2): rare for good reason — easy to strip accidentally, painful to cache against.

Deprecation is where the gateway earns its keep. A sunset flow that works in practice: route /api/v1/* to the legacy upstream, attach Sunset and Deprecation response headers at the edge, emit a deprecated_routes counter per client ID, and contact the stragglers before flipping the route to 410 Gone. Because the traffic data already lives at the gateway, you know exactly which clients still call the old version — no survey emails, no guessing. The mistake to avoid is keeping v1 alive “just in case” with no headers and no metrics: versions never die on their own, they die when you can see who still uses them.

Production Scenario: Kong for E-commerce

A concrete reference architecture: six services (users, catalog, orders, payments, inventory, notifications) behind Kong on Kubernetes, with Keycloak for OIDC and Redis for rate limiting. The middleware pipeline from this guide — auth, token bucket, cache, circuit breaker over a prefix router — is runnable from the companion repository.

Kong configuration (kong.yml):
  services:
    - name: user-service
      url: http://user-service.default.svc:8080
      routes:
        - name: users-route
          paths: ["/api/users"]
          strip_path: true
      plugins:
        - name: jwt
        - name: rate-limiting
          config: { minute: 100, hour: 1000 }
        - name: correlation-id
          config: { header_name: "X-Request-ID" }

Authentication flow (Keycloak OIDC):
  1. Client gets JWT: POST /realms/ecommerce/protocol/openid-connect/token
  2. Client sends: Authorization: Bearer <token>
  3. Kong validates JWT with the realm public key
  4. Validation result cached in Redis (TTL: 5 min)
  5. Valid → routed to service; invalid → 401 at the edge

Distributed rate limiting:
  kong.conf: rate_limiting = redis (redis.default.svc:6379)
  Tiers: Free 100/min · Pro 1.000/min · Enterprise 10.000/min

Observability:
  - Prometheus plugin → /metrics
  - Correlation-id on every request for distributed traces
  - Grafana: QPS, p50/p95/p99 latency, error rate per route

Kubernetes deployment:
  - Kong via Helm: 3 replicas on different nodes
  - HPA: min 3, max 10, CPU target 70%
  - PodDisruptionBudget: minAvailable 2
  - Health: /status endpoint · Graceful shutdown: 30s drain

For an architecture this size, illustrative SLO targets look like: gateway-added latency under 2ms at p95, auth cache hit rate above 90%, error rate under 0.1% — treat these as goals to instrument against, not guaranteed results.

Deployment Patterns

The gateway itself must not be the single point of failure you built it to eliminate. Three replicas minimum, spread across nodes (or availability zones), behind a load balancer:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: gateway
          image: registry.io/api-gateway:latest
          ports:
            - containerPort: 8080
          resources:
            limits: { cpu: 1000m, memory: 512Mi }
            requests: { cpu: 500m, memory: 256Mi }
          livenessProbe:
            httpGet: { path: /health, port: 8080 }
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5

Details that matter more than the YAML: separate /health (liveness — process alive) from /ready (readiness — config loaded, upstreams reachable); drain connections on shutdown so in-flight requests finish; keep a PodDisruptionBudget so node maintenance never takes all replicas at once; and version your declarative config in git — Kong’s deck or Envoy’s xDS both treat config as deployable artifacts that should roll out like code.

Best Practices

• Keep the gateway stateless — session data in Redis or client-side tokens, never in gateway memory.
• Use async I/O and connection pooling to upstreams to avoid thread exhaustion and TCP handshake overhead.
• Propagate request IDs injected at the edge through every service for distributed tracing.
• Cache authentication results (with short TTLs) so every request doesn’t hit the identity provider.
• Split internal vs external gateways when auth schemes and rate limits diverge — one size fits nobody.
• Set request size limits at the edge so oversized payloads never reach service memory.
• Treat gateway config as code — versioned, reviewed, and rolled out with the same discipline as application deploys.

Common Mistakes

• Business logic in the gateway: pricing rules and domain validation in gateway plugins create a distributed monolith that deploys in lockstep.
• Per-instance rate limiting: local counters mean a client bypasses limits by hitting different replicas — always shared state.
• Missing upstream timeouts: one slow service exhausts gateway threads and starves healthy routes.
• Retries on non-idempotent methods: retrying writes without idempotency keys duplicates side effects.
• Unversioned breaking changes: retire API versions with Sunset headers and months of notice, not a 404 one morning.
• Ignoring gateway metrics: the gateway is infrastructure on the critical path — it needs the same alerting rigor as your databases.

When to Use

• Migrating from a monolith to microservices and needing a unified entry point — pairs naturally with the strangler fig approach.
• Clients talking directly to services and duplicating auth, retries, and logging in every SDK.
• Cascading failures, inconsistent auth, or observability blind spots across services.
• Multiple client types (public API, partners, internal tools) that need different policies at the edge.

When NOT to Use

• A monolith or a couple of services — the gateway is an extra hop and failure domain for zero benefit.
• Internal service-to-service calls — that’s what a service mesh or direct mTLS is for.
• Teams that can’t operate the gateway itself — a badly run gateway is worse than no gateway; managed options (AWS API Gateway, Apigee) trade control for operational burden.

Troubleshooting

• Requests bypass rate limits intermittently: classic per-instance counter bug — the client lands on different replicas. Move counters to Redis and verify all replicas share the store.
• Sudden 401s after a key rotation: the gateway cached the old JWKS or validation result too long. Shorten auth cache TTL during rotations and fetch keys from the JWKS endpoint, not static config.
• Latency added by the gateway spikes under load: look for synchronous chains — sequential auth call, then rate limit check, then cache lookup. Parallelize independent stages and use async I/O.
• Headers disappear between gateway and service: aggressive allowlist/denylist filters strip X-* headers. Audit transform plugins; the usual suspects are a default “strip unknown headers” rule.
• One flaky upstream takes everything down: circuit breaker thresholds too lenient or timeouts missing. Set per-service timeouts and verify the breaker opens under a chaos test, not just in the config file.
• Retries amplify an outage instead of absorbing it: retries with no jitter land as synchronized waves on a recovering service. Cap the retry count at 1–2, add exponential backoff with jitter, and set a global retry budget so the gateway never multiplies load beyond a few percent of baseline.
• Config changes reach replicas at different times: rolling a route change across replicas one by one leaves clients hitting mixed behavior for minutes. Push config through a versioned pipeline and gate the rollout on health checks, or accept it consciously for non-breaking changes only.

Further Reading

• Kong Gateway documentation — plugin reference for auth, rate limiting, and proxy-cache used in the scenario.
• Envoy documentation — the filter model behind the routing example and modern cloud-native gateways.
• Azure Architecture Center: API gateways — vendor-neutral guidance on gateway responsibilities and BFF splits.
• Microservices architecture guide — where the gateway fits in the broader architecture.
• Gateway routing pattern — the pattern-level companion to this guide.
• Runnable companion — the middleware pipeline (auth, rate limit, cache, circuit breaker) with tests.