Distributed Consensus: Raft and Leader Election
Concept
Any system with multiple nodes that needs to agree on a single, ordered sequence of facts — who's the leader, what order writes happened in, what the current configuration is — runs into the same problem: nodes can fail, messages can be delayed or lost, and yet the group still has to converge on one agreed answer, not several conflicting ones. That's the distributed consensus problem, and it's the mechanism underneath every system that promises strong consistency across replicas: etcd, Kubernetes' control plane, CockroachDB, Consul, and Kafka's KRaft mode (which replaced its old ZooKeeper dependency) are all Raft implementations under the hood.
Raft was explicitly designed to be an understandable alternative to Paxos, which solves the same problem but is notoriously difficult to reason about or implement correctly. Raft decomposes consensus into two separable pieces:
Leader election. Every node is a follower, candidate, or leader. Followers expect periodic heartbeats (AppendEntries RPCs) from a leader; if none arrive before a randomized election timeout, a follower becomes a candidate, increments a monotonic term number, and requests votes from every other node. A candidate that wins a majority becomes leader for that term — and randomizing the timeout is the trick that keeps multiple nodes from perpetually splitting the vote by starting elections in lockstep.
Log replication. Once elected, the leader is the only node that accepts writes. Every write becomes a log entry the leader replicates to followers via AppendEntries; an entry is only considered committed — safe to apply and acknowledge to the client — once a majority of nodes have durably stored it. This majority requirement is what makes consensus tolerate failures: a 5-node cluster keeps functioning correctly as long as any 3 nodes are up, because a majority-of-3 write is guaranteed to overlap with the majority-of-3 needed for the next leader to have seen it — that overlap guarantee is the core safety property, called the Leader Completeness property.
Tradeoffs
| Property | What you get | What it costs |
|---|---|---|
| Strong consistency via majority quorum | Every committed write survives leader failure and can't be silently lost | Every write pays the latency of a round trip to a majority of nodes, not just the leader |
| Single-leader writes | Simple, unambiguous ordering — no write conflicts to resolve | The leader is a throughput ceiling; all writes serialize through one node until it changes |
| Odd cluster sizes (3, 5, 7) | Maximizes fault tolerance per node added — 5 nodes tolerates 2 failures, same as 6 nodes would | Diminishing returns past 5-7 nodes — more nodes means more replication traffic per write for the same fault tolerance an odd count already gives you |
| Randomized election timeouts | Prevents repeated split-vote elections | A leader failure still costs one full election-timeout window of unavailability for writes, by design |
The fundamental trade is availability-for-writes versus correctness: Raft chooses to make the cluster briefly unavailable for new writes during a leader election rather than risk two leaders in the same term accepting conflicting writes (a split-brain). Systems that instead prioritize always-available writes over consistency (e.g., Dynamo-style multi-leader replication) push the conflict-resolution problem to read time instead, trading it for eventual consistency and merge conflicts.
When to use / when not to
- Reach for a Raft-based coordination service (etcd, Consul, ZooKeeper) when you need a small amount of strongly-consistent shared state — leader election for your own service, distributed locks, configuration that every node must agree on — not as a general-purpose database.
- Building consensus into an application-level database is almost never the right call directly; use an existing Raft/Paxos implementation (etcd, or a database that embeds one, like CockroachDB or TiDB) rather than hand-rolling leader election — this is one of the most failure-prone things to reimplement from scratch.
- Don't put high-throughput, high-volume data through a Raft-backed store — every write pays majority round-trip latency, so it's suited for control-plane state (config, membership, locks, leader pointers), not for a system's primary high-volume data path.
- Skip distributed consensus entirely for single-writer or single-instance state — it only earns its cost once multiple nodes genuinely need to agree on something despite failures.
Common pitfall
Assuming a Raft cluster with fewer than a working majority of nodes is merely degraded rather than fully unavailable for writes. A 5-node cluster that loses 3 nodes doesn't run at "40% capacity" — it stops accepting writes entirely, because no majority can be formed to commit anything, even though 2 nodes are still up and reachable. Capacity planning for a Raft-backed system has to account for this cliff, not a gradual degradation curve — losing your majority is a hard outage for writes, not a performance hit.
Engineering Lens
The Principal-level distinction to draw in a design review isn't "do we need consensus" — it's "which piece of our system's state actually needs strong agreement, and which piece can tolerate eventual consistency instead." Most systems have a small control-plane surface (leader pointers, configuration, membership, distributed locks) that genuinely needs Raft-style guarantees, sitting next to a much larger data-plane surface that doesn't. Conflating the two — either by putting bulk data through a consensus store, or by trying to get away with a non-consensus mechanism for something that actually needs it (a naive "first writer wins" lock without quorum) — is a common architecture mistake. Being able to point at the exact piece of state that needs consensus, and defend why the rest doesn't, reads as someone who understands the cost being paid rather than someone reaching for etcd because it's the default answer to "distributed state."