Architecture/Fundamentals/health-checks-liveness-vs-readiness

Health Checks: Liveness vs Readiness Probes

Concept

"Is this instance healthy?" is actually two different questions that get conflated into a single /health endpoint far too often. Liveness asks: is this process still running and not deadlocked/hung? If liveness fails, the answer is drastic — kill and restart the process, because nothing short of a restart is going to fix a genuinely wedged instance. Readiness asks a different question: is this instance currently able to serve traffic correctly right now? An instance can be perfectly alive (process running fine) but not ready — still warming up a cache, waiting on a database connection pool to establish, or mid-way through a graceful shutdown drain — and in all of those cases, restarting it would be the wrong response; the correct response is simply to stop routing new traffic to it until it's ready again.

Kubernetes formalizes this split explicitly (livenessProbe and readinessProbe), but the underlying pattern predates it and applies to any load-balanced fleet: a load balancer's health check is functionally a readiness check (pull instances that can't serve traffic out of rotation), while a process supervisor's health check is functionally a liveness check (restart what's actually stuck). Conflating the two — using one check for both purposes — means either a hung process keeps receiving traffic because "readiness" logic doesn't know to declare it dead, or a slow-starting-but-fine instance gets killed and restarted in a crash loop because "liveness" logic mistook temporary unreadiness for being stuck.

Tradeoffs

Approach	Failure handled correctly	Failure handled wrong
Single combined health check, binary pass/fail	Simple total outage of the instance	Warming-up instance killed and restarted (should've just waited); hung instance kept in rotation (should've been killed)
Separate liveness + readiness, liveness triggers restart	Deadlocked/hung process	None if implemented correctly, but a bad liveness check (too aggressive) can restart a healthy-but-slow instance, causing self-inflicted churn
Separate liveness + readiness, readiness gates traffic only	Instance not ready for traffic (startup, dependency down, draining)	A hung process that still returns 200 on liveness never gets restarted — liveness logic has to actually detect the hang, not just "process is running"

The tension is precision-of-signal versus complexity: a single check is trivial to implement but conflates two failure modes that need opposite remedies (restart vs. pull-from-rotation-and-wait); two checks require more code and more careful definition of what "alive" actually means beyond "the process didn't crash," but they let the orchestrator apply the correct remedy to each failure mode instead of guessing.

When to use / when not to

Always separate the two for anything running under an orchestrator that can act on both signals differently (Kubernetes, ECS, most modern platforms) — the split is nearly free once the orchestrator supports it.
Readiness checks should verify real dependencies the instance actually needs to serve a request correctly (database reachable, cache warmed, downstream auth service reachable) — not just "process is up," which is what liveness already covers.
Liveness checks should be conservative and cheap — checking that the event loop/request-handling thread isn't wedged, not re-verifying every downstream dependency (a flaky downstream dependency failing a liveness check causes a restart storm that doesn't fix the actual problem).
During deploys, readiness gating during startup and during graceful shutdown drain is what prevents a rolling deploy from routing live traffic to instances that aren't ready yet or are already shutting down.
Skip the distinction only for genuinely stateless, instantly-ready processes with no startup dependency — a trivial static-content server has little to gain from the split.

Common pitfall

Making the liveness check call out to a downstream dependency (a database, an external API). If that dependency has a transient blip, every instance's liveness check fails simultaneously, and the orchestrator restarts the entire fleet at once — the health-check design itself becomes the outage. Liveness should answer "is this process stuck," not "are all of this process's dependencies currently healthy" — that second question belongs to readiness, which pulls the instance from rotation without killing it, a far safer response to a dependency blip.

Principal Engineer Lens

Health-check design is a small piece of configuration that encodes a real architectural decision about failure response, and it's an easy thing to get subtly wrong in a way that only shows up under a specific failure mode — a downstream blip that causes a fleet-wide restart storm is exactly the kind of self-inflicted incident that's hard to explain in a postmortem ("the health check caused the outage, not the actual dependency issue"). Being able to state, for a given service, exactly what its liveness check verifies and exactly what its readiness check verifies — and why those are different — is a quick, concrete signal of operational maturity in a review.

Sources: