Architecture/Fundamentals/distributed-locking-and-fencing-tokens

Distributed Locking and Fencing Tokens

Concept

A distributed lock is meant to guarantee that only one process, across an entire fleet of machines, is doing a particular piece of work at a time — sending an email, writing to a shared file, updating a row that isn't protected by a database transaction. The common implementation is a lease on a coordination store: acquire a key in Redis or ZooKeeper with a TTL, do the work, release it. It looks like a mutex. It is not one.

The failure mode that makes this dangerous is a process pause. A client acquires the lock, then stalls — a GC pause, a CPU steal on a noisy-neighbor VM, a slow disk I/O, a network partition — for longer than the lease's TTL. The lock service, seeing no renewal, expires the lease and hands it to another client. The first client eventually wakes up, has no idea time has passed, and — believing it still holds the lock — proceeds to write to the shared resource. Now two clients think they're the exclusive owner, and there is no way for the lock service to prevent this: it cannot know whether the first client is dead or merely slow, and a distributed system can never reliably distinguish the two.

Martin Kleppmann's well-known 2016 critique of Redis's Redlock algorithm made this precise: no amount of "more nodes voting" fixes a fundamentally unsafe protocol, because the danger isn't losing quorum, it's clock and pause assumptions that don't hold under real-world GC pauses and scheduling delays. His fix is a fencing token: every time the lock is granted, the lock service hands out a monotonically increasing number along with it. The client includes that token with every write to the protected resource, and the resource itself — not the lock service — rejects any write carrying a token lower than the highest one it has already seen. If client A's lease expires and client B acquires the lock with token 34 while client A (still believing it holds token 33) tries to write, the storage layer rejects A's stale write. The lock stops being "trust me, I have it" and becomes an ordering primitive the resource enforces.

Tradeoffs

Mechanism	Guarantees	Cost
Single-node lock (Redis `SET NX`)	Fast, simple	No fault tolerance — that node's failure loses the lock's safety
Redlock (multi-node Redis quorum)	Survives node failure	Still unsafe under process pauses/clock skew per Kleppmann's critique — treat as a fast-path optimization, not a correctness guarantee
Consensus-backed lease (ZooKeeper, etcd, Consul)	Strong ordering via a real consensus protocol (Raft/ZAB)	Higher latency per acquire (majority round trip); still needs fencing tokens for full safety
Fencing tokens at the resource	Actually safe — invalid stale writes are rejected at the point of enforcement	Requires the protected resource to understand and check tokens; doesn't help if the resource can't be modified (e.g., an external API call)

The uncomfortable truth for all of these: a lock service alone, no matter how consistent, cannot make a distributed system correct if the protected action isn't validated where it happens. The lock is advisory; the fencing token (or an equivalent idempotency check) is what's enforced.

When to use / when not to

Use for coordinating infrequent, coarse-grained work — leader election, "only one instance runs this cron job," preventing a background reconciler from double-processing.
Always pair the lock with a fencing token or equivalent idempotency check at the resource being protected, whenever the resource can enforce one (a database row with a version column, an object store with conditional writes).
When the protected resource can't check a token (e.g., calling a third-party payment API), prefer designing the operation to be naturally idempotent (see Idempotency Keys) rather than leaning on the lock alone.
Don't reach for a distributed lock as a substitute for a database transaction or a proper consensus store (see Distributed Consensus: Raft and Leader Election) when the work is naturally scoped to a single database's ACID guarantees.
Don't treat lock acquisition as proof of exclusivity for anything where a stale write causes real damage (financial postings, inventory decrements) unless fencing is enforced end to end.

Common pitfall

Assuming a lock with a TTL is safe because "the TTL will expire it eventually." The TTL solves liveness (the lock won't be held forever if a client crashes) but does nothing for the safety problem: a paused-then-resumed client doesn't know it lost the lock, and will act as if it still owns it. Bumping the TTL up doesn't fix this either — it only changes how long the system stays vulnerable before another client can take over, while making the underlying race no less real. The only real fix is making the resource itself reject stale operations via a fencing token or equivalent monotonic check.

Principal Engineer Lens

This is one of the cleanest examples of the gap between "looks correct" and "is correct" in distributed systems, and it's worth being able to walk through in a design review from first principles: a lock is a belief held by a client, and beliefs can be wrong once real-world pauses enter the picture — GC, scheduler preemption, network jitter. The Principal-level move is recognizing that safety has to be pushed down to the resource being protected, not upheld by trusting whichever client currently thinks it owns the lock. This generalizes past locking specifically: it's the same reasoning behind idempotency keys, optimistic concurrency control (version columns), and conditional writes — anywhere a "believe the caller" protocol needs to become a "verify at the point of truth" protocol. In Fintech and payments contexts especially, a stale-write bug from an over-trusted lock is exactly the kind of subtle correctness gap that turns into a duplicate settlement or a double-spend, so being fluent in fencing tokens (and knowing Redlock's known limitations) is a real credibility signal in review.