Circuit Breaker Pattern
Concept
In a distributed system, a slow or failing downstream dependency is more dangerous than one that fails fast — callers pile up threads/connections waiting on it, exhaust their own resource pools, and the failure cascades upstream into services that were themselves healthy. The circuit breaker pattern, popularized by Martin Fowler (building on Michael Nygard's Release It!), wraps a call to a protected dependency in an object that tracks failures and, once they cross a threshold, stops making the call entirely — failing fast instead of piling up.
The breaker moves through three states:
- Closed — normal operation, calls pass through, failures are counted.
- Open — failure threshold exceeded; calls are rejected immediately (no network call made at all) for a cooldown period.
- Half-Open — after the cooldown, a limited number of trial calls are let through; success closes the breaker again, failure re-opens it.
Tradeoffs
| State handling | Benefit | Cost |
|---|---|---|
| No breaker (always call through) | Simplest, always tries | Cascading failure — one slow dependency exhausts caller threads/connections and takes down callers too |
| Circuit breaker | Fails fast once a dependency is known-bad, protects caller's own resources | Adds statefulness and tuning surface (thresholds, cooldown duration) to every call site; a too-sensitive breaker trips on transient blips and adds unnecessary failures |
| Timeout alone (no breaker) | Simpler than a breaker | Doesn't stop repeated attempts against a dependency that's known to be down — still wastes resources retrying a call that's near-certain to fail |
A breaker is not a substitute for a timeout — it complements one. The timeout bounds a single call; the breaker remembers the pattern across calls and stops making them at all once that pattern says "don't bother."
When to use / when not to
- Use wherever a service calls another service (or external dependency) over a network and that dependency's failure could cascade — service-to-service calls, third-party API integrations, database connection pools under load.
- Especially valuable at fan-out points: a single upstream service calling many downstream services benefits most, since one bad downstream shouldn't be allowed to starve calls to the healthy ones.
- Less useful for calls with no meaningful "known-bad" state to detect — a one-off batch job with no repeated calls to the same dependency doesn't get much value from a breaker's failure-tracking.
- Don't reach for it as the first fix for a slow dependency — first ask why it's slow; a breaker manages the symptom (cascading load) but doesn't fix the root cause.
Common pitfall
Setting the failure threshold and cooldown by guesswork instead of the dependency's actual behavior under load, which produces one of two failure modes: a breaker that's too sensitive (trips on ordinary transient blips, adding availability loss the dependency itself never had), or one that's too lenient (stays closed long enough that cascading failure happens anyway before it trips). Both require real load-testing or production tuning, not a default library value left untouched.
Engineering Lens
Circuit breakers are a concrete instance of a broader distributed-systems instinct: design for partial failure as the normal case, not the exception. The pattern's real value in a design review isn't "we added a circuit breaker library" — it's being able to say what happens to the caller when a specific downstream dependency is fully down, and showing that answer was designed rather than discovered during an incident. This reasoning applies identically whether the downstream is a payments processor in a Fintech stack, a market-data feed in a trading system, or an internal network-device API — the failure-isolation logic doesn't change with the domain.