Build Resilient Systems with the Circuit Breaker Pattern
How to prevent cascading failures in distributed systems using circuit breakers with open, closed, and half-open states in Java, TypeScript, and Python.
Overview
A microservice calls a downstream payment service. Say the payment service starts crawling because its database is struggling. The calling service spawns more threads, each waiting for the payment service to respond. Thread pools saturate. From then on the service refuses new calls until it gets healthy again. The services that depend on it also fail. Within minutes, a localized database slowdown has brought down the entire request chain. That’s a cascading failure.
A circuit breaker sits in the middle and watches every call to that downstream service. If the failure rate exceeds a threshold, the circuit “opens” and further calls fail fast without reaching the struggling service. After a timeout, the circuit enters a “half-open” state and allows a test request through. Success closes the circuit back up. This gives the failing service time to recover and prevents the caller from wasting resources on doomed requests. Here is how to state machine design, implementation, and integration with retry and fallback strategies.
When to use it
Use this recipe when:
- Calling external services over a network where failures are inevitable
- Preventing a slow downstream service from consuming all caller resources
- Providing fast failure with fallback instead of blocking on timeouts
- Protecting thread pools, connection pools, and memory from being exhausted
- Building resilient microservice architectures where partial failures are contained
Solution
TypeScript Circuit Breaker
type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';
interface CircuitBreakerConfig {
failureThreshold: number; // failures before opening
recoveryTimeout: number; // ms before half-open
halfOpenMaxCalls: number; // test calls in half-open
}
class CircuitBreaker<T extends (...args: any[]) => Promise<any>> {
private state: CircuitState = 'CLOSED';
private failures = 0;
private successes = 0;
private nextAttempt = Date.now();
constructor(
private fn: T,
private config: CircuitBreakerConfig
) {}
async execute(...args: Parameters<T>): Promise<ReturnType<T>> {
if (this.state === 'OPEN') {
if (Date.now() < this.nextAttempt) {
throw new Error('Circuit breaker is OPEN');
}
this.state = 'HALF_OPEN';
this.successes = 0;
}
try {
const result = await this.fn(...args);
this.onSuccess();
return result;
} catch (error) {
this.onFailure();
throw error;
}
}
private onSuccess(): void {
if (this.state === 'HALF_OPEN') {
this.successes++;
if (this.successes >= this.config.halfOpenMaxCalls) {
this.state = 'CLOSED';
this.failures = 0;
}
} else {
this.failures = 0;
}
}
private onFailure(): void {
this.failures++;
if (this.state === 'HALF_OPEN' || this.failures >= this.config.failureThreshold) {
this.state = 'OPEN';
this.nextAttempt = Date.now() + this.config.recoveryTimeout;
}
}
getState(): CircuitState {
return this.state;
}
}
// Usage
const paymentBreaker = new CircuitBreaker(
(amount: number) => paymentService.charge(amount),
{ failureThreshold: 5, recoveryTimeout: 30000, halfOpenMaxCalls: 3 }
);
try {
await paymentBreaker.execute(100);
} catch (error) {
await fallbackPaymentProcessor.charge(100);
}
Java with Resilience4j
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry;
public class PaymentService {
private final CircuitBreaker circuitBreaker;
public PaymentService() {
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
.failureRateThreshold(50)
.waitDurationInOpenState(Duration.ofMillis(30000))
.permittedNumberOfCallsInHalfOpenState(3)
.slidingWindowSize(10)
.build();
CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);
this.circuitBreaker = registry.circuitBreaker("payment");
}
public String charge(double amount) {
return circuitBreaker.executeSupplier(() -> {
return externalPaymentGateway.charge(amount);
});
}
}
Python with pybreaker
import pybreaker
import requests
class PaymentService:
def __init__(self):
self.breaker = pybreaker.CircuitBreaker(
fail_max=5,
reset_timeout=30,
expected_exception=requests.RequestException
)
@pybreaker.circuit
def charge(self, amount):
response = requests.post(
"https://payment-api.example.com/charge",
json={"amount": amount}
)
response.raise_for_status()
return response.json()
service = PaymentService()
try:
result = service.charge(100)
except pybreaker.CircuitBreakerError:
result = fallback_charge(100)
Explanation
- Closed state: the circuit is closed and requests pass through to the downstream service. Failures are counted in a sliding window. Cross that threshold — rate or count — and the circuit pops open. In the closed state, a small number of failures is tolerated — networks are inherently unreliable.
- Open state: the circuit is open and all requests fail fast with a
CircuitBreakerOpenexception (or similar). The downstream service isn’t called, preventing resource exhaustion. The circuit stays open for a configured recovery timeout (e.g., 30 seconds), giving the failing service time to heal. - Half-open state: after the recovery timeout, the circuit transitions to half-open. A handful of probe requests get through to test the waters. When the probes come back clean, the circuit closes again. Any failure pops it back open and restarts the timeout. That automatic probe saves you from babysitting the recovery by hand.
- Sliding window: the breaker counts failures over the last 10 calls or 60 seconds, not since startup. This prevents a circuit from opening due to a single spike if the aggregate rate is healthy. And once failures stop, the design gives the service room to recover on its own.
Variants
| Type | Failure detection | Recovery | Best for |
|---|---|---|---|
| Count-based | N failures in window | Fixed timeout | Steady load, predictable failure patterns |
| Time-based | N failures in duration | Adaptive timeout | Variable load, burst traffic |
| Percentage | >X% failure rate | Timeout | Large volume where absolute count is noisy |
| Custom metric | Latency, error rate | Manual | Complex systems with multiple health signals |
What Works
- Always provide a fallback: when the circuit is open, the application must still function. A payment API can answer “pending — try again shortly”; a catalog can quietly serve slightly stale cache data. Never let an open circuit propagate as a hard failure to the user.
- Use circuit breakers with timeouts and retries: a circuit breaker without a per-request timeout can still hang. Combine a circuit breaker (macro-level health) with a request timeout (micro-level limit) and retry (transient recovery). The retry should be inside the circuit breaker, not outside — retrying on an open circuit is wasted effort.
- Log and alert on circuit state changes: opening a circuit is a symptom of a downstream problem. Log every state transition with context (failure rate, last error, affected service). Alert when a circuit opens, but suppress recovery alerts unless the circuit repeatedly opens — a single recovery is normal; a flapping circuit isn’t.
- Size the sliding window to your traffic: a window of 10 calls opens after 5 failures, which is appropriate for moderate traffic. On high-throughput services a percentage window works better — trip at 50% failures over the last 100 calls, for example. For low-traffic services, a count-based window may never accumulate enough failures — consider time-based windows instead.
- Distinguish between failure types: don’t open the circuit on 4xx errors (client errors that will not recover). Trip only on 5xx, timeouts, and connection errors — never on business-logic 4xx. Some libraries allow configuring
expected_exception— use it to classify non-retryable errors.
Common mistakes
- Wrapping every call in a circuit breaker: breakers aren’t free — each one keeps state, counters, and timers. Save it for calls that cross a network or service boundary — anything that can hang, time out, or go sideways. Do not wrap in-memory, in-process function calls — they don’t fail in ways a circuit breaker helps.
- Ignoring the half-open state: some breakers jump straight from open to closed when the timeout fires. This is dangerous — if the service is still failing, the circuit immediately reopens and you get a loop of open-close-open. Don’t snap the circuit shut — let a few trial calls through and see how they go.
- Setting recovery timeout too short: a 1-second timeout on a database that needs 30 seconds to fail over causes rapid flapping. Set the recovery timeout based on observed recovery times. Downstream needs 2 minutes to restart? Give the timeout 2.5, so the circuit doesn’t probe a service that’s still booting.
- Not monitoring circuit metrics: skip the metrics and you’ll never know how often circuits trip or how long they stay tripped. Export circuit state, failure counts, and open duration to Prometheus, CloudWatch, or Datadog. Use dashboards to spot systemic issues.
Further Reading
- Martin Fowler — CircuitBreaker — the canonical write-up on the pattern
- Resilience4j documentation — Java implementation with sliding-window metrics
- Polly resilience policies — the .NET circuit breaker library
- Release It! — Michael Nygard — the book that named the pattern
Frequently Asked Questions
Is a circuit breaker just a fancy if-statement?
No — it's a state machine with memory (failure windows), automatic recovery (half-open), and distributed coordination. A simple if (failures > 5) throw doesn't recover automatically, doesn't track sliding windows, and doesn't allow controlled test requests.
Should I retry inside or outside the circuit breaker?
Retry inside. The circuit breaker wraps the retry logic. If the retry exhausts and the call still fails, the circuit breaker counts it as a failure. Retrying outside an open circuit wastes resources — you already know the service is unhealthy.
Can circuit breakers cause data inconsistency?
Yes — when the fallback is sloppy. If the circuit opens during a payment and the fallback is "assume it succeeded," you may mark unpaid orders as paid. Design fallbacks to be safe: mark as pending, queue for later processing, or notify the user. See Saga Pattern for distributed transaction coordination.
How do circuit breakers work with async/await?
They handle it for you — Resilience4j, Opossum for JS, and Polly all wrap async calls natively. The state machine runs in the calling thread (or event loop), and the wrapped function is awaited. Timeouts must be compatible with the async runtime (Promise timeout in JS, CompletableFuture timeout in Java). See Async Patterns for async execution strategies.
Related Resources
Microservices Resilience Patterns
How to build fault-tolerant distributed systems using circuit breakers, bulkheads, retries with backoff, and sagas for transaction management.
RecipeManage Distributed Transactions with the Saga Pattern
How to implement saga orchestration and choreography to maintain data consistency across microservices without distributed transactions or two-phase commit.
RecipeDesign a Scalable API Gateway for Microservices
Build an API gateway that routes requests, handles authentication, rate limiting, caching, and protocol translation between clients and backend microservices.
RecipeRetry Logic with Exponential Backoff
How to implement resilient retry logic with exponential backoff and jitter for transient failures in network and API calls.
RecipeMulti-Tenancy Architecture
Serve many tenants from one deployment with the right isolation level: shared tables with row-level security, schema-per-tenant, or database-per-tenant.
RecipeService Discovery
Implement service discovery with health checks, DNS-based resolution, and service registries for live microservices environments.