API Gateway Design: Resilience, Routing, and Security

Overview

Every microservices architecture eventually needs a front door. The API gateway is that door — and if you design it poorly, it becomes a single point of failure, a performance bottleneck, or a security gap. A well-designed gateway handles routing, rate limiting, authentication, circuit breaking, and observability so that individual services can focus on business logic. Here is a hands-on guide to the core patterns, trade-offs, and implementation strategies for building resilient API gateways.

When to Use

• For alternatives, see Complete Guide to API Security.

Use this guide when:

• You are migrating from a monolith to microservices and need a unified entry point
• Your client applications talk directly to backend services and you need to consolidate cross-cutting concerns
• You are experiencing cascading failures, authentication inconsistencies, or observability blind spots across services

Solution

Core Gateway Responsibilities

Concern	Implementation Approach	Key Technology
Request Routing	Path-based, header-based, or host-based routing to upstream services	Nginx, Kong, Envoy, AWS API Gateway
Rate Limiting	Token bucket or leaky bucket per client, IP, or API key	Redis + Lua, Envoy, Kong
Authentication	Validate JWT, API keys, or OAuth2 tokens at the edge	Keycloak, Auth0, custom middleware
Circuit Breaker	Track failure rates and fail fast when upstreams are unhealthy	Resilience4j, Envoy, Hystrix (legacy)
Load Balancing	Round-robin, least connections, or consistent hashing	Envoy, Nginx, service mesh
Observability	Distributed tracing, metrics, and access logs	OpenTelemetry, Prometheus, ELK
SSL Termination	Offload TLS at the gateway to reduce CPU load on services	Nginx, HAProxy, cloud load balancers
Request/Response Transformation	Header injection, payload rewriting, protocol translation	Kong plugins, Envoy filters

Routing Architecture Example

# Envoy configuration snippet
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/api/users" }
                          route: { cluster: user_service, prefix_rewrite: "/" }
                        - match: { prefix: "/api/orders" }
                          route: { cluster: order_service, prefix_rewrite: "/" }
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: user_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: user_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address: { address: user-service, port_value: 8080 }
    - name: order_service
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: order_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address: { address: order-service, port_value: 8080 }

Rate Limiting Implementation

# Token bucket rate limiter with Redis
import time
import redis

class TokenBucketRateLimiter:
    def __init__(self, redis_client, key_prefix, capacity, refill_rate):
        self.redis = redis_client
        self.key_prefix = key_prefix
        self.capacity = capacity
        self.refill_rate = refill_rate

    def is_allowed(self, client_id):
        key = f"{self.key_prefix}:{client_id}"
        pipeline = self.redis.pipeline()
        now = time.time()
        pipeline.hmget(key, ["tokens", "last_refill"])
        result = pipeline.execute()
        tokens = float(result[0][0] or self.capacity)
        last_refill = float(result[0][1] or now)

        elapsed = now - last_refill
        tokens = min(self.capacity, tokens + elapsed * self.refill_rate)

        if tokens >= 1:
            tokens -= 1
            self.redis.hmset(key, {"tokens": tokens, "last_refill": now})
            self.redis.expire(key, 60)
            return True
        else:
            self.redis.hmset(key, {"tokens": tokens, "last_refill": last_refill})
            return False

Circuit Breaker Pattern

// Resilience4j circuit breaker configuration
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)
    .slowCallRateThreshold(80)
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .permittedNumberOfCallsInHalfOpenState(10)
    .slidingWindowSize(100)
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .build();

CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);
CircuitBreaker userServiceCB = registry.circuitBreaker("userService");

Supplier<String> decorated = CircuitBreaker
    .decorateSupplier(userServiceCB, () -> userClient.getUser(userId));

Explanation

The API gateway is a reverse proxy with intelligence. It sits between clients and backend services, centralizing cross-cutting concerns that would otherwise be duplicated in every service. The key insight is that not every concern belongs in the gateway: business logic should stay in services, but authentication, rate limiting, and routing are gateway responsibilities.

Routing decisions should be stateless and deterministic so that any gateway instance can handle any request. Session affinity (sticky sessions) should be avoided; if you need it, prefer consistent hashing over IP-based stickiness. Rate limiting must be distributed in a multi-instance gateway deployment; local in-memory counters are insufficient because a client can round-robin across instances and bypass limits.

Circuit breakers prevent cascading failures. When a service is struggling, the gateway should fail fast rather than queue requests that will timeout. This protects both the struggling service and the caller. The half-open state allows gradual recovery: after a cooldown, a limited number of requests are allowed through to test if the service has recovered.

Detailed Scenario: API Gateway with Kong for E-commerce

System: E-commerce with 6 microservices behind Kong
Services: users, catalog, orders, payments, inventory, notifications
Platform: Kubernetes (EKS) + Kong + Redis + Keycloak

Kong configuration (kong.yml):
  services:
    - name: user-service
      url: http://user-service.default.svc:8080
      routes:
        - name: users-route
          paths: ["/api/users"]
          strip_path: true
      plugins:
        - name: jwt
        - name: rate-limiting
          config: { minute: 100, hour: 1000 }
        - name: correlation-id
          config: { header_name: "X-Request-ID" }

    - name: order-service
      url: http://order-service.default.svc:8080
      routes:
        - name: orders-route
          paths: ["/api/orders"]
          strip_path: true
      plugins:
        - name: jwt
        - name: rate-limiting
          config: { minute: 50, hour: 500 }
        - name: correlation-id

Authentication with Keycloak (OIDC):
  1. Client obtains JWT from Keycloak: POST /realms/ecommerce/protocol/openid-connect/token
  2. Client sends JWT in header: Authorization: Bearer <token>
  3. Kong validates JWT with Keycloak public key
  4. Kong caches validation result in Redis (TTL: 5 min)
  5. If valid, Kong routes to service; if not, returns 401

Distributed rate limiting with Redis:
  kong.conf:
    rate_limiting = redis
    redis_host = redis.default.svc
    redis_port = 6379
    redis_timeout = 2000

  Limits per plan:
    Free: 100 req/min, 1000 req/hour
    Pro: 1000 req/min, 10000 req/hour
    Enterprise: 10000 req/min, no hourly limit

Observability:
  - Prometheus plugin: exports metrics to /metrics
  - Zipkin plugin: distributed traces with correlation ID
  - File-log plugin: JSON logs to stdout (collected by Fluentd)
  - Grafana dashboard: QPS, latency p50/p95/p99, error rate per route

Circuit breaker per service:
  kong.yml plugin:
    - name: proxy-cache
      config: { strategy: "redis", cache_ttl: 300 }
    - name: pre-function
      functions: [check_upstream_health]

  Logic: if error rate > 50% in 30s, open circuit for 60s
  When open: return 503 with Retry-After header
  When half-open: allow 10 probe requests

Kubernetes deployment:
  - Kong via Helm chart: 3 replicas on different nodes
  - HPA: min 3, max 10 replicas, CPU target 70%
  - PodDisruptionBudget: min available 2
  - Health check: /status endpoint
  - Graceful shutdown: 30s drain period

Production metrics (average):
  | Metric | Value |
  |--------|-------|
  | Total QPS | 12,000 |
  | Gateway p50 latency | 1.2ms |
  | Gateway p95 latency | 3.8ms |
  | Auth cache hit rate | 94% |
  | Error rate | 0.08% |
  | Uptime | 99.97% |

How do I handle API versioning in the gateway?

Use path-based versioning: /v1/users, /v2/users. The gateway routes to different services or versions of the same service. Keep v1 running while v2 is in use. Schedule v1 deprecation with a response header: Sunset: Sat, 31 Dec 2026 23:59:59 GMT. Document changes between versions. Never break v1 without 6 months notice.

Variants

Gateway Type	Best For	Trade-off
Nginx / OpenResty	High throughput, simple routing	Lua scripting for custom logic; limited plugin platform
Kong	Plugin-rich, enterprise capabilities	Higher latency than Nginx; managed option available
Envoy	Cloud-native, service mesh integration	Steep learning curve; YAML configuration verbosity
AWS API Gateway	Serverless, AWS platform	Vendor lock-in; cold start latency for v2
Spring Cloud Gateway	Java/Spring platforms	JVM memory footprint; tight Spring coupling
Traefik	Dynamic Docker/K8s discovery	Simpler than Envoy; less advanced traffic management

What Works

1. Keep the gateway stateless; store session data in Redis or client-side tokens
2. Use async I/O in the gateway to avoid thread pool exhaustion under load
3. Implement health checks for every upstream and remove unhealthy instances promptly
4. Log request IDs and propagate them via headers for distributed tracing
5. Cache authentication results at the gateway to avoid validating the same token on every request

Common Mistakes

1. Putting business logic in the gateway; it becomes a distributed monolith
2. Using the gateway as a shared database connection pool; services should manage their own connections
3. Implementing rate limiting per instance instead of distributed; clients bypass limits by hitting different instances
4. Not setting timeouts on upstream calls; a slow service can exhaust gateway threads
5. Ignoring gateway health in monitoring; the gateway is infrastructure, not invisible plumbing

Troubleshooting

• High latency between services: trace the request path. Look for synchronous chains, missing caching, and oversized payloads that cross network boundaries.
• Single point of failure: identify components without redundancy. Add replicas, failover, or circuit breakers before scaling traffic.
• Unexpected coupling between services: review shared databases, libraries, and schemas. Bound contexts should own their data and expose stable interfaces.
• Cost spikes after scaling: right-size instances and use autoscaling with limits. Reserved capacity or spot instances can reduce steady-state spend.
• Difficult to reason about the system: maintain architecture decision records and service dependency maps. Use observability to validate the diagrams.

FAQ

Should I use one gateway or multiple?

One gateway is simpler but becomes a bottleneck and a single point of failure. Most organizations eventually split by concern: a public gateway for external clients, an internal gateway for service-to-service communication, and possibly a B2B gateway for partner APIs. The rule of thumb: split when your gateway configuration exceeds 1,000 lines or when different clients need fundamentally different authentication schemes.

How do I handle WebSocket connections through a gateway?

WebSockets require connection-aware routing. The gateway must maintain the TCP connection and route subsequent frames to the same upstream instance. Not all gateways support this natively. Envoy and Nginx do; AWS API Gateway v2 supports WebSockets with Lambda or HTTP integrations. For high-scale WebSocket workloads, consider a dedicated WebSocket gateway separate from your HTTP API gateway.

What is the performance impact of adding a gateway?

A well-tuned gateway adds 0.5–2ms latency per hop for simple routing. SSL termination can actually improve total latency because services no longer do TLS handshakes. The biggest risk is misconfiguration: overly complex regex routing, synchronous blocking I/O, or excessive request/response transformation. Benchmark your gateway independently using tools like k6 or vegeta before deploying to production.

Common Production Pitfalls

• Treating the guide as a checklist to complete once rather than a practice to evolve.
• Adopting every recommendation at once instead of starting with one measured change.
• Skipping the maturity assessment and forcing advanced practices on an unprepared team.
• Not updating runbooks and on-call expectations as new practices are introduced.
• Ignoring real incident data when prioritizing which parts of the guide to apply first.
• Failing to assign an owner who reviews decisions quarterly.
• Copying examples without adapting them to the team’s actual tooling and constraints.
• Forgetting to measure outcomes before adding the next improvement.