Architecture/Fundamentals/read-replicas-and-replication-lag

Read Replicas and Replication Lag

Concept

Most application workloads are read-heavy — far more queries fetch data than write it. A single primary database eventually can't serve that read volume alongside writes without contention, so the standard scaling move is to add read replicas: read-only copies of the primary that continuously receive a stream of changes and can serve read traffic independently, letting reads scale out horizontally while writes stay centralized on the primary.

The mechanism underneath is log-based replication. The primary writes every change to a durability log before applying it (PostgreSQL's write-ahead log/WAL, MySQL's binlog); replicas connect to the primary, stream that log, and replay each entry to reconstruct the same state. This is inherently asynchronous by default — the primary acknowledges a write to the client as soon as it's durable on the primary itself, without waiting for any replica to catch up. That asynchrony is exactly what makes replicas cheap to add (no write-latency penalty as you scale reads out) and exactly what creates replication lag: the gap between when a write commits on the primary and when a replica has replayed it and reflects it in query results. Lag is typically milliseconds under normal load, but it is not bounded — a replica falling behind on hardware, network, or a long-running query on the replica itself can stretch that gap to seconds or minutes with no automatic ceiling.

Lag isn't a corner case to design around defensively — it's the direct, structural cost of the scaling technique. A client that writes a row and then immediately reads it back from a replica can get a stale (or missing) result, a pattern called read-your-writes inconsistency, and it is one of the single most common sources of "it works on retry" bugs in systems that reach for read replicas without accounting for it.

Tradeoffs

Consistency model	Read scalability	Staleness risk	Complexity
Read everything from primary	None — primary is still the bottleneck	Zero	Lowest
Async replicas, no staleness handling	High — reads scale near-linearly with replica count	Read-your-writes bugs whenever a client reads its own recent write from a replica	Low to add, but the staleness bugs are subtle and often found in production
Async replicas + sticky routing (route a session's reads to the primary right after it writes)	High for the general case, slightly reduced during the sticky window	Bounded to the specific session that just wrote, not all readers	Moderate — needs session/request-level routing logic
Synchronous or semi-synchronous replication	Lower — primary now waits on replica acknowledgment before committing	None (or near-none)	Highest — adds write latency and a new availability dependency on replica health

The core trade is between write latency and read-freshness guarantees: async replication keeps writes fast and cheap by not waiting on replicas at all, at the cost of an unbounded (if usually small) staleness window; synchronous replication closes that window but makes every write pay the latency and availability cost of a healthy replica responding — the same latency-versus-durability tension that shows up in sharding and in quorum-based writes generally.

When to use / when not to

Use read replicas as the default first move to scale a read-heavy workload past what a single primary can serve — it's cheaper and simpler than sharding, and doesn't require partitioning the data model.
Route reads that must reflect a just-completed write (an order confirmation page right after checkout, a settings page right after a save) to the primary, or use sticky/session-based routing that pins a client to the primary briefly after it writes — don't send every read blindly to "any replica."
Use replicas for genuinely read-only, staleness-tolerant workloads first — dashboards, reporting, search indexing, analytics — where a few hundred milliseconds of lag is invisible to the use case.
Don't reach for synchronous replication unless the business actually requires zero-staleness reads at all times (financial ledger balances, inventory counts at the moment of sale) — the write-latency and availability cost isn't worth paying for workloads that can tolerate eventual consistency.
Don't treat replica count as an unlimited scaling knob — every replica adds load to the primary's replication stream, and past a certain fan-out the primary itself becomes the bottleneck for keeping replicas current, not just for serving writes.

Common pitfall

Assuming replication lag is bounded and small because it usually is under normal conditions, then discovering in an incident that a replica silently fell hours behind — often because a long-running analytical query held a lock or consumed I/O on the replica, or because the replica's network path degraded — while the application kept routing reads to it as if it were current. The fix isn't just monitoring lag (though that's necessary); it's building the application layer to actually check replica lag against a threshold and fail over reads to the primary, or reject serving from a replica whose lag exceeds an acceptable bound, rather than trusting that lag will always stay small because it usually does.

Principal Engineer Lens

The question worth pressing in a design review isn't "do we have read replicas" — nearly every system at scale does — it's "which specific reads in this system require freshness, and how does the routing layer guarantee they get it." Most teams add replicas reactively (the primary got slow) without ever auditing which read paths are staleness-sensitive, and the read-your-writes bugs that follow get discovered by confused users and support tickets, not code review. Being able to name the exact reads that must hit the primary — and defend why the rest are fine on a replica with a stated lag tolerance — is the difference between "we scaled reads" and "we understand the consistency model we actually ship." That distinction matters most exactly where staleness has real cost: a trading platform showing a stale position, or a payments dashboard showing a balance that hasn't caught up with a transaction the user just completed.

Reel Script

Setup: A user submits a payment, gets redirected to a confirmation page, and the confirmation page says the payment doesn't exist — then refreshing five seconds later shows it fine. Nothing was actually broken; the read just hit a replica that hadn't caught up yet.

Concept walkthrough: Explain log-based replication — the primary streams its write-ahead log to replicas, which replay it to reconstruct state — and why this is asynchronous by default: the primary doesn't wait for a replica before acknowledging a write, which is exactly what makes replicas cheap to scale out and exactly what creates a lag window.

Real example tie-in: Walk through PostgreSQL streaming replication on something like RDS — WAL segments stream continuously to each replica, and lag is measured as the gap between the primary's latest commit and what the replica has actually replayed, which can spike under replica-side load even when the primary is healthy.

Tradeoffs & alternatives: Lay out the spectrum from "everything reads the primary" (safe, doesn't scale) to full synchronous replication (safe, costs write latency) with async replicas plus sticky/primary-routing for freshness-sensitive reads as the pragmatic middle most systems actually land on.

Principal Engineer takeaway: The senior move isn't adding replicas — it's auditing which specific read paths in the system are staleness-sensitive and routing only those to the primary, rather than either blindly trusting every replica or over-correcting by avoiding replicas altogether.

Sources: