Load Balancing Algorithms
Concept
A load balancer's job sounds simple — spread requests across a fleet of backend servers — but the algorithm it uses to pick the next server changes both throughput and correctness in ways that only show up under real traffic patterns, not synthetic even load.
- Round Robin — cycle through the server list in fixed order, one request each. Trivial to implement, works well for a homogeneous fleet handling roughly uniform-cost requests, but it's blind to actual server load — a server still working through a slow request gets the next one anyway.
- Weighted Round Robin — the same rotation, but each server gets a weight proportional to its capacity, so a bigger instance receives proportionally more requests. Fixes round robin's blindness to heterogeneous fleets, but the weights are still static, not reactive to real-time load.
- Least Connections — route to whichever server currently has the fewest active connections. Adapts to real load and handles variable-cost requests well (a server stuck on a slow request naturally stops receiving new ones), at the cost of needing the load balancer to track live connection counts per backend.
- Least Response Time — least connections plus a live latency signal, routing to the server with the best combination of low connection count and fast recent response times. More adaptive still, at the cost of more state to track and more tuning to get the combination right.
- Consistent Hashing — hash the request key (often a client ID, session ID, or cache key) to deterministically map it to the same backend server every time, and do so in a way that adding or removing a server only remaps a small fraction of keys instead of reshuffling everything. This is the mechanism that makes distributed caches and sharded stores viable — it's covered in depth in Database Sharding Strategies because sharding is an application of consistent hashing, not a separate technique.
- Random / Power of Two Choices — pick two servers at random and route to whichever has fewer active connections. Nearly matches full least-connections accuracy at a fraction of the coordination cost, which matters when the load balancer is one of many nodes that don't share a global view of every backend's load in real time.
Tradeoffs
| Algorithm | Adapts to real-time load | Handles heterogeneous servers | Session/cache affinity | Coordination cost |
|---|---|---|---|---|
| Round Robin | No | No (without weighting) | No | None |
| Weighted Round Robin | No (static weights) | Yes | No | Low — weights set once |
| Least Connections | Yes | Yes | No | Needs live per-backend connection counts |
| Least Response Time | Yes, most reactive | Yes | No | Needs live connection + latency tracking |
| Consistent Hashing | No (by design — same key, same server) | Partially, via virtual nodes | Yes — this is the point | Needs a shared hash ring, low rebalancing cost on scale change |
| Power of Two Choices | Approximately | Yes | No | Low — only samples 2 servers, doesn't need full global state |
The real fork in the road is between algorithms optimizing for even load (round robin variants, least connections, power of two choices) and consistent hashing, which deliberately sacrifices load evenness to guarantee the same key always lands on the same server — a tradeoff that only makes sense when affinity (cache hit rate, session stickiness, shard ownership) matters more than perfectly balanced load. Picking a load-optimizing algorithm for a cache tier defeats the cache; picking consistent hashing for a stateless API tier adds coordination overhead for an affinity guarantee nothing needs.
When to use / when not to
- Use round robin (or weighted round robin for a heterogeneous fleet) as the default for stateless services with roughly uniform request cost — it's the cheapest option and there's no load-shape problem for it to solve.
- Use least connections or least response time when request cost varies significantly — long-running queries mixed with fast reads, for instance — so the load balancer actually reacts to which servers are backed up.
- Use consistent hashing wherever affinity matters more than perfectly even load: distributed caches (so a cache key always hits the node that has it warm), sharded databases, and session-stateful services that keep in-memory state per user.
- Use power of two choices at large fleet scale, where tracking exact global connection counts across every load balancer node is itself a coordination cost — it gets most of least-connections' benefit without that overhead.
- Don't reach for consistent hashing by default "because it's the sophisticated choice" — for a genuinely stateless fleet it adds hash-ring coordination for zero benefit over round robin.
Common pitfall
Rebalancing a hash ring naively when the backend fleet scales up or down — without virtual nodes, adding or removing even one real server can remap a large fraction of keys at once, which for a cache tier means a sudden wave of cache misses hitting the origin simultaneously (a self-inflicted thundering herd triggered by routine autoscaling, not by traffic). Virtual nodes — mapping each physical server to many points on the hash ring — are what keep consistent hashing's rebalancing cost proportional to 1/N instead of remapping everything.
Engineering Lens
The algorithm choice itself is rarely the hard part of a design review — it's recognizing which of the two families (load-optimizing vs. affinity-preserving) the workload actually needs, and being able to say why in one sentence: "this tier is stateless, round robin is fine" versus "this tier is a cache, we need consistent hashing or the cache doesn't work." The same reasoning shows up in Capital Markets-adjacent systems under a different name — order routing across matching engines or liquidity venues has to choose between spreading load evenly and preserving affinity (routing related orders to the same venue for consistency), which is the identical tension consistent hashing solves for caches, just with financial instead of infrastructure stakes.