Load Balancing Algorithms
Concept
A load balancer's job sounds simple — spread requests across a fleet of backend servers — but the algorithm it uses to pick the next server changes both throughput and correctness in ways that only show up under real traffic patterns, not synthetic even load.
- Round Robin — cycle through the server list in fixed order, one request each. Trivial to implement, works well for a homogeneous fleet handling roughly uniform-cost requests, but it's blind to actual server load — a server still working through a slow request gets the next one anyway.
- Weighted Round Robin — the same rotation, but each server gets a weight proportional to its capacity, so a bigger instance receives proportionally more requests. Fixes round robin's blindness to heterogeneous fleets, but the weights are still static, not reactive to real-time load.
- Least Connections — route to whichever server currently has the fewest active connections. Adapts to real load and handles variable-cost requests well (a server stuck on a slow request naturally stops receiving new ones), at the cost of needing the load balancer to track live connection counts per backend.
- Least Response Time — least connections plus a live latency signal, routing to the server with the best combination of low connection count and fast recent response times. More adaptive still, at the cost of more state to track and more tuning to get the combination right.
- Consistent Hashing — hash the request key (often a client ID, session ID, or cache key) to deterministically map it to the same backend server every time, and do so in a way that adding or removing a server only remaps a small fraction of keys instead of reshuffling everything. This is the mechanism that makes distributed caches and sharded stores viable — it's covered in depth in Database Sharding Strategies because sharding is an application of consistent hashing, not a separate technique.
- Random / Power of Two Choices — pick two servers at random and route to whichever has fewer active connections. Nearly matches full least-connections accuracy at a fraction of the coordination cost, which matters when the load balancer is one of many nodes that don't share a global view of every backend's load in real time.
Tradeoffs
| Algorithm | Adapts to real-time load | Handles heterogeneous servers | Session/cache affinity | Coordination cost |
|---|---|---|---|---|
| Round Robin | No | No (without weighting) | No | None |
| Weighted Round Robin | No (static weights) | Yes | No | Low — weights set once |
| Least Connections | Yes | Yes | No | Needs live per-backend connection counts |
| Least Response Time | Yes, most reactive | Yes | No | Needs live connection + latency tracking |
| Consistent Hashing | No (by design — same key, same server) | Partially, via virtual nodes | Yes — this is the point | Needs a shared hash ring, low rebalancing cost on scale change |
| Power of Two Choices | Approximately | Yes | No | Low — only samples 2 servers, doesn't need full global state |
The real fork in the road is between algorithms optimizing for even load (round robin variants, least connections, power of two choices) and consistent hashing, which deliberately sacrifices load evenness to guarantee the same key always lands on the same server — a tradeoff that only makes sense when affinity (cache hit rate, session stickiness, shard ownership) matters more than perfectly balanced load. Picking a load-optimizing algorithm for a cache tier defeats the cache; picking consistent hashing for a stateless API tier adds coordination overhead for an affinity guarantee nothing needs.
When to use / when not to
- Use round robin (or weighted round robin for a heterogeneous fleet) as the default for stateless services with roughly uniform request cost — it's the cheapest option and there's no load-shape problem for it to solve.
- Use least connections or least response time when request cost varies significantly — long-running queries mixed with fast reads, for instance — so the load balancer actually reacts to which servers are backed up.
- Use consistent hashing wherever affinity matters more than perfectly even load: distributed caches (so a cache key always hits the node that has it warm), sharded databases, and session-stateful services that keep in-memory state per user.
- Use power of two choices at large fleet scale, where tracking exact global connection counts across every load balancer node is itself a coordination cost — it gets most of least-connections' benefit without that overhead.
- Don't reach for consistent hashing by default "because it's the sophisticated choice" — for a genuinely stateless fleet it adds hash-ring coordination for zero benefit over round robin.
Common pitfall
Rebalancing a hash ring naively when the backend fleet scales up or down — without virtual nodes, adding or removing even one real server can remap a large fraction of keys at once, which for a cache tier means a sudden wave of cache misses hitting the origin simultaneously (a self-inflicted thundering herd triggered by routine autoscaling, not by traffic). Virtual nodes — mapping each physical server to many points on the hash ring — are what keep consistent hashing's rebalancing cost proportional to 1/N instead of remapping everything.
Principal Engineer Lens
The algorithm choice itself is rarely the hard part of a design review — it's recognizing which of the two families (load-optimizing vs. affinity-preserving) the workload actually needs, and being able to say why in one sentence: "this tier is stateless, round robin is fine" versus "this tier is a cache, we need consistent hashing or the cache doesn't work." The same reasoning shows up in Capital Markets-adjacent systems under a different name — order routing across matching engines or liquidity venues has to choose between spreading load evenly and preserving affinity (routing related orders to the same venue for consistency), which is the identical tension consistent hashing solves for caches, just with financial instead of infrastructure stakes.
Reel Script
Setup: A caching layer sits in front of a slow origin database. Someone points a plain round-robin load balancer at the cache nodes to "spread the load evenly," and cache hit rates immediately crater — the same key now lands on a different node every time, so nothing ever stays warm.
Concept walkthrough: Walk through the two families side by side: load-optimizing algorithms (round robin, weighted round robin, least connections, power of two choices) that try to spread requests evenly, versus consistent hashing, which deliberately routes the same key to the same server every time at the cost of even load. Explain why that tradeoff is the whole point for a cache or a sharded store.
Real example tie-in: Trace what actually broke in the setup scenario — round robin sent the same cache key to five different nodes across five requests, so all five missed and hit the origin, exactly the load the cache existed to absorb. Swapping to consistent hashing (with virtual nodes to keep rebalancing cheap) fixes it by guaranteeing key-to-node stability.
Tradeoffs & alternatives: Name the fleet-scale nuance too — least connections needs a global view of every backend's load, which gets expensive to coordinate at large scale, and power of two choices is the pragmatic middle ground that gets close to the same result by sampling just two servers instead of tracking all of them.
Principal Engineer takeaway: Before naming an algorithm, name the requirement it needs to satisfy — even load, or key affinity — because the two families solve different problems, and the wrong pick for the tier in question doesn't fail loudly, it just quietly degrades (cache hit rate, tail latency) in a way that's easy to miss until someone goes looking.
Related
Sources: