Circuit Breaker Pattern
Concept
In a distributed system, a slow or failing downstream dependency is more dangerous than one that fails fast — callers pile up threads/connections waiting on it, exhaust their own resource pools, and the failure cascades upstream into services that were themselves healthy. The circuit breaker pattern, popularized by Martin Fowler (building on Michael Nygard's Release It!), wraps a call to a protected dependency in an object that tracks failures and, once they cross a threshold, stops making the call entirely — failing fast instead of piling up.
The breaker moves through three states:
- Closed — normal operation, calls pass through, failures are counted.
- Open — failure threshold exceeded; calls are rejected immediately (no network call made at all) for a cooldown period.
- Half-Open — after the cooldown, a limited number of trial calls are let through; success closes the breaker again, failure re-opens it.
Tradeoffs
| State handling | Benefit | Cost |
|---|---|---|
| No breaker (always call through) | Simplest, always tries | Cascading failure — one slow dependency exhausts caller threads/connections and takes down callers too |
| Circuit breaker | Fails fast once a dependency is known-bad, protects caller's own resources | Adds statefulness and tuning surface (thresholds, cooldown duration) to every call site; a too-sensitive breaker trips on transient blips and adds unnecessary failures |
| Timeout alone (no breaker) | Simpler than a breaker | Doesn't stop repeated attempts against a dependency that's known to be down — still wastes resources retrying a call that's near-certain to fail |
A breaker is not a substitute for a timeout — it complements one. The timeout bounds a single call; the breaker remembers the pattern across calls and stops making them at all once that pattern says "don't bother."
When to use / when not to
- Use wherever a service calls another service (or external dependency) over a network and that dependency's failure could cascade — service-to-service calls, third-party API integrations, database connection pools under load.
- Especially valuable at fan-out points: a single upstream service calling many downstream services benefits most, since one bad downstream shouldn't be allowed to starve calls to the healthy ones.
- Less useful for calls with no meaningful "known-bad" state to detect — a one-off batch job with no repeated calls to the same dependency doesn't get much value from a breaker's failure-tracking.
- Don't reach for it as the first fix for a slow dependency — first ask why it's slow; a breaker manages the symptom (cascading load) but doesn't fix the root cause.
Common pitfall
Setting the failure threshold and cooldown by guesswork instead of the dependency's actual behavior under load, which produces one of two failure modes: a breaker that's too sensitive (trips on ordinary transient blips, adding availability loss the dependency itself never had), or one that's too lenient (stays closed long enough that cascading failure happens anyway before it trips). Both require real load-testing or production tuning, not a default library value left untouched.
Principal Engineer Lens
Circuit breakers are a concrete instance of a broader distributed-systems instinct: design for partial failure as the normal case, not the exception. The pattern's real value in a design review isn't "we added a circuit breaker library" — it's being able to say what happens to the caller when a specific downstream dependency is fully down, and showing that answer was designed rather than discovered during an incident. This reasoning applies identically whether the downstream is a payments processor in a Fintech stack, a market-data feed in a trading system, or an internal network-device API — the failure-isolation logic doesn't change with the domain.
Reel Script
Setup: Picture a service that calls a downstream API, and that API starts timing out. Without any protection, every caller thread now sits there waiting on a 30-second timeout, one after another, until the caller itself runs out of threads — a downstream outage just became an upstream outage too.
Concept walkthrough: Introduce the three states — closed (normal, calls pass, failures counted), open (threshold hit, reject immediately without even trying the network call), half-open (after a cooldown, let a few trial calls through to check if it's recovered). Emphasize the open state's whole point: fail fast, don't fail slow.
Real example tie-in: Walk a fan-out scenario — one API gateway calling five backend services, one of which is down. Without a breaker, calls to the healthy four services can get starved because threads are stuck waiting on the broken one. With a breaker on that one dependency, it trips, calls to it fail instantly, and the other four keep serving normally.
Tradeoffs & alternatives: Contrast with timeouts alone — a timeout bounds one call, a breaker remembers the pattern across many calls. Note the tuning cost: thresholds and cooldowns set by guesswork produce either a jumpy breaker or a too-slow one, both worse than a well-tuned one.
Principal Engineer takeaway: The strong review answer isn't "we use Resilience4j/Hystrix" — it's naming exactly what happens to your own service's resource pool when a specific dependency goes fully down, and showing that was a deliberate design decision, not something discovered live during an incident.
Related
Sources: