StackPractices
intermediate By Mathias Paulenko

Build Resilient Systems with the Circuit Breaker Pattern

How to prevent cascading failures in distributed systems using circuit breakers with open, closed, and half-open states in Java, TypeScript, and Python.

Overview

A microservice calls a downstream payment service. Say the payment service starts crawling because its database is struggling. The calling service spawns more threads, each waiting for the payment service to respond. Thread pools saturate. From then on the service refuses new calls until it gets healthy again. The services that depend on it also fail. Within minutes, a localized database slowdown has brought down the entire request chain. That’s a cascading failure.

A circuit breaker sits in the middle and watches every call to that downstream service. If the failure rate exceeds a threshold, the circuit “opens” and further calls fail fast without reaching the struggling service. After a timeout, the circuit enters a “half-open” state and allows a test request through. Success closes the circuit back up. This gives the failing service time to recover and prevents the caller from wasting resources on doomed requests. Here is how to state machine design, implementation, and integration with retry and fallback strategies.

When to use it

Use this recipe when:

  • Calling external services over a network where failures are inevitable
  • Preventing a slow downstream service from consuming all caller resources
  • Providing fast failure with fallback instead of blocking on timeouts
  • Protecting thread pools, connection pools, and memory from being exhausted
  • Building resilient microservice architectures where partial failures are contained

Solution

TypeScript Circuit Breaker

type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

interface CircuitBreakerConfig {
  failureThreshold: number;      // failures before opening
  recoveryTimeout: number;       // ms before half-open
  halfOpenMaxCalls: number;      // test calls in half-open
}

class CircuitBreaker<T extends (...args: any[]) => Promise<any>> {
  private state: CircuitState = 'CLOSED';
  private failures = 0;
  private successes = 0;
  private nextAttempt = Date.now();

  constructor(
    private fn: T,
    private config: CircuitBreakerConfig
  ) {}

  async execute(...args: Parameters<T>): Promise<ReturnType<T>> {
    if (this.state === 'OPEN') {
      if (Date.now() < this.nextAttempt) {
        throw new Error('Circuit breaker is OPEN');
      }
      this.state = 'HALF_OPEN';
      this.successes = 0;
    }

    try {
      const result = await this.fn(...args);
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess(): void {
    if (this.state === 'HALF_OPEN') {
      this.successes++;
      if (this.successes >= this.config.halfOpenMaxCalls) {
        this.state = 'CLOSED';
        this.failures = 0;
      }
    } else {
      this.failures = 0;
    }
  }

  private onFailure(): void {
    this.failures++;
    if (this.state === 'HALF_OPEN' || this.failures >= this.config.failureThreshold) {
      this.state = 'OPEN';
      this.nextAttempt = Date.now() + this.config.recoveryTimeout;
    }
  }

  getState(): CircuitState {
    return this.state;
  }
}

// Usage
const paymentBreaker = new CircuitBreaker(
  (amount: number) => paymentService.charge(amount),
  { failureThreshold: 5, recoveryTimeout: 30000, halfOpenMaxCalls: 3 }
);

try {
  await paymentBreaker.execute(100);
} catch (error) {
  await fallbackPaymentProcessor.charge(100);
}

Java with Resilience4j

import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry;

public class PaymentService {
    private final CircuitBreaker circuitBreaker;

    public PaymentService() {
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
            .failureRateThreshold(50)
            .waitDurationInOpenState(Duration.ofMillis(30000))
            .permittedNumberOfCallsInHalfOpenState(3)
            .slidingWindowSize(10)
            .build();

        CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);
        this.circuitBreaker = registry.circuitBreaker("payment");
    }

    public String charge(double amount) {
        return circuitBreaker.executeSupplier(() -> {
            return externalPaymentGateway.charge(amount);
        });
    }
}

Python with pybreaker

import pybreaker
import requests

class PaymentService:
    def __init__(self):
        self.breaker = pybreaker.CircuitBreaker(
            fail_max=5,
            reset_timeout=30,
            expected_exception=requests.RequestException
        )

    @pybreaker.circuit
    def charge(self, amount):
        response = requests.post(
            "https://payment-api.example.com/charge",
            json={"amount": amount}
        )
        response.raise_for_status()
        return response.json()

service = PaymentService()

try:
    result = service.charge(100)
except pybreaker.CircuitBreakerError:
    result = fallback_charge(100)

Explanation

stateDiagram-v2 diagram: [*
  • Closed state: the circuit is closed and requests pass through to the downstream service. Failures are counted in a sliding window. Cross that threshold — rate or count — and the circuit pops open. In the closed state, a small number of failures is tolerated — networks are inherently unreliable.
  • Open state: the circuit is open and all requests fail fast with a CircuitBreakerOpen exception (or similar). The downstream service isn’t called, preventing resource exhaustion. The circuit stays open for a configured recovery timeout (e.g., 30 seconds), giving the failing service time to heal.
  • Half-open state: after the recovery timeout, the circuit transitions to half-open. A handful of probe requests get through to test the waters. When the probes come back clean, the circuit closes again. Any failure pops it back open and restarts the timeout. That automatic probe saves you from babysitting the recovery by hand.
  • Sliding window: the breaker counts failures over the last 10 calls or 60 seconds, not since startup. This prevents a circuit from opening due to a single spike if the aggregate rate is healthy. And once failures stop, the design gives the service room to recover on its own.

Variants

TypeFailure detectionRecoveryBest for
Count-basedN failures in windowFixed timeoutSteady load, predictable failure patterns
Time-basedN failures in durationAdaptive timeoutVariable load, burst traffic
Percentage>X% failure rateTimeoutLarge volume where absolute count is noisy
Custom metricLatency, error rateManualComplex systems with multiple health signals

What Works

  • Always provide a fallback: when the circuit is open, the application must still function. A payment API can answer “pending — try again shortly”; a catalog can quietly serve slightly stale cache data. Never let an open circuit propagate as a hard failure to the user.
  • Use circuit breakers with timeouts and retries: a circuit breaker without a per-request timeout can still hang. Combine a circuit breaker (macro-level health) with a request timeout (micro-level limit) and retry (transient recovery). The retry should be inside the circuit breaker, not outside — retrying on an open circuit is wasted effort.
  • Log and alert on circuit state changes: opening a circuit is a symptom of a downstream problem. Log every state transition with context (failure rate, last error, affected service). Alert when a circuit opens, but suppress recovery alerts unless the circuit repeatedly opens — a single recovery is normal; a flapping circuit isn’t.
  • Size the sliding window to your traffic: a window of 10 calls opens after 5 failures, which is appropriate for moderate traffic. On high-throughput services a percentage window works better — trip at 50% failures over the last 100 calls, for example. For low-traffic services, a count-based window may never accumulate enough failures — consider time-based windows instead.
  • Distinguish between failure types: don’t open the circuit on 4xx errors (client errors that will not recover). Trip only on 5xx, timeouts, and connection errors — never on business-logic 4xx. Some libraries allow configuring expected_exception — use it to classify non-retryable errors.

Common mistakes

  • Wrapping every call in a circuit breaker: breakers aren’t free — each one keeps state, counters, and timers. Save it for calls that cross a network or service boundary — anything that can hang, time out, or go sideways. Do not wrap in-memory, in-process function calls — they don’t fail in ways a circuit breaker helps.
  • Ignoring the half-open state: some breakers jump straight from open to closed when the timeout fires. This is dangerous — if the service is still failing, the circuit immediately reopens and you get a loop of open-close-open. Don’t snap the circuit shut — let a few trial calls through and see how they go.
  • Setting recovery timeout too short: a 1-second timeout on a database that needs 30 seconds to fail over causes rapid flapping. Set the recovery timeout based on observed recovery times. Downstream needs 2 minutes to restart? Give the timeout 2.5, so the circuit doesn’t probe a service that’s still booting.
  • Not monitoring circuit metrics: skip the metrics and you’ll never know how often circuits trip or how long they stay tripped. Export circuit state, failure counts, and open duration to Prometheus, CloudWatch, or Datadog. Use dashboards to spot systemic issues.

Further Reading

Frequently Asked Questions

Is a circuit breaker just a fancy if-statement?

No — it's a state machine with memory (failure windows), automatic recovery (half-open), and distributed coordination. A simple if (failures > 5) throw doesn't recover automatically, doesn't track sliding windows, and doesn't allow controlled test requests.

Should I retry inside or outside the circuit breaker?

Retry inside. The circuit breaker wraps the retry logic. If the retry exhausts and the call still fails, the circuit breaker counts it as a failure. Retrying outside an open circuit wastes resources — you already know the service is unhealthy.

Can circuit breakers cause data inconsistency?

Yes — when the fallback is sloppy. If the circuit opens during a payment and the fallback is "assume it succeeded," you may mark unpaid orders as paid. Design fallbacks to be safe: mark as pending, queue for later processing, or notify the user. See Saga Pattern for distributed transaction coordination.

How do circuit breakers work with async/await?

They handle it for you — Resilience4j, Opossum for JS, and Polly all wrap async calls natively. The state machine runs in the calling thread (or event loop), and the wrapped function is awaited. Timeouts must be compatible with the async runtime (Promise timeout in JS, CompletableFuture timeout in Java). See Async Patterns for async execution strategies.