Developer/Availability/FaultTolerance/Fundamentals/failure-detection-and-split-brain-avoidance

Failure Detection and Split-Brain Avoidance

Concept

Fault tolerance depends on a step that's easy to take for granted: before a system can fail over to a backup, it has to correctly decide the primary is actually dead. That decision is harder than it looks, because a network partition is indistinguishable from a real failure to the node on the other side of it — a primary that's merely cut off from its standby by a flaky link is, from the standby's point of view, identical to a primary that's crashed. Getting this wrong in either direction is expensive: fail over too eagerly on a false positive and both nodes can end up believing they're primary at once (split-brain), each accepting writes independently; fail over too slowly on a real failure and the outage runs longer than it needed to.

The standard building block is a heartbeat — a periodic liveness signal nodes exchange (or send to a shared coordinator) so that a missed heartbeat past some timeout triggers a failure suspicion. Heartbeats alone don't solve split-brain, though, because a partition breaks the heartbeat channel exactly the same way a crash does. The fix is quorum: a failover decision is only acted on if a majority of the cluster's nodes agree the primary is unreachable, not just the one standby watching it. An odd-sized cluster (3, 5, 7 nodes) or a dedicated lightweight witness/arbiter node ensures there's always a decisive majority, so a 2-way network split can't produce two partitions that both believe they hold quorum. Even with quorum, an isolated old primary might come back online still believing it's primary — the final safeguard is fencing, commonly implemented as STONITH ("Shoot The Other Node In The Head"): before the new primary starts accepting writes, the cluster forcibly powers off, reboots, or network-isolates the old one, guaranteeing it cannot re-emerge and issue conflicting writes.

Tradeoffs

Mechanism	What it catches	What it doesn't	Cost
Heartbeat only	Real crashes (heartbeat genuinely stops)	Network partitions with primary still alive — triggers false failover	Cheapest, simplest
Heartbeat + quorum	Most partitions — a minority side can't unilaterally promote itself	A true majority-side partition can still starve the minority of service (correct, but users on that side see an outage)	Needs an odd node count or a witness; added cluster-membership complexity
Heartbeat + quorum + fencing (STONITH)	The remaining case: an old primary that reappears after being outvoted	Nothing structural — this is the complete safety story for this failure class	Highest — needs a real out-of-band power/network control path to the node being fenced

The tension is between failover speed and failover safety. A short heartbeat timeout catches real failures fast but is more prone to tripping on transient network blips; a long timeout is safer against false positives but extends real-outage duration. Quorum and fencing don't remove that tuning tension — they exist to make sure that whichever way the timeout gets tuned, the system never ends up with two active primaries at once, which is a correctness violation, not just an availability one.

When to use / when not to

Use quorum-based failover for any stateful service where two simultaneously-active primaries would corrupt data — primary-replica databases, distributed locks, leader-elected coordinators (etcd, ZooKeeper, Consul all build on this exact model).
Skip formal quorum machinery for stateless services behind a load balancer — a load balancer's health check removing an unresponsive instance from rotation is a much simpler failure-detection problem, since there's no shared-write-conflict risk if a "dead" instance briefly comes back.
Add fencing (STONITH or equivalent) specifically when the cost of a resurrected old primary writing data is severe (financial ledgers, inventory counts) — for a cache or read-replica, a stale node briefly rejoining is usually a much smaller problem than the operational cost of a real fencing mechanism.
Don't rely on a single witness or arbiter as if it were a full quorum member with no failure mode of its own — a witness that's itself unreachable during the real partition just moves the tie-breaking problem rather than solving it; managed consensus systems (etcd/Raft, ZooKeeper/ZAB) are usually a better answer than a hand-rolled witness node.

Common pitfall

Deploying a 2-node "high availability" pair and treating it as fault-tolerant. Two nodes can never form a quorum-safe majority on their own — a network partition between them leaves each side with exactly 50%, and no mechanism inside a bare 2-node setup can safely decide which side should become primary without an external tie-breaker. This is why production HA clusters use 3+ voting members or an explicit external witness; a 2-node pair without one is one network blip away from split-brain, not protected against it.

Engineering Lens

The question worth asking about any "automatic failover" claim in a design review is: what specifically prevents two nodes from both believing they're primary during a partition? A confident answer names the quorum size and the fencing mechanism explicitly; a vague answer ("the standby takes over if the primary stops responding") is describing heartbeat-only detection, which is exactly the setup that produces split-brain under partition rather than crash. This distinction matters more, not less, as systems move to multi-region deployments, where cross-region network partitions are common enough to be a normal operating condition rather than a rare edge case — a failover design that's only ever been tested against clean node crashes hasn't been tested against its actual failure mode in production.