Hermes Wiki

Expand-Contract Pattern for Zero-Downtime Schema Changes

Concept

A schema change that renames a column, changes a type, or drops a table is a breaking change for any application code deployed at the moment the migration runs. If the migration and the code deploy aren't perfectly simultaneous — and in a rolling deploy across multiple instances, they never are — some instances read/write the old shape while others expect the new one, and requests fail in between. The expand-contract pattern (also called parallel change, a term Martin Fowler uses on his bliki) avoids this by never changing something in place. Instead it splits one breaking change into a sequence of non-breaking ones, each independently deployable and independently safe to roll back:

  1. Expand — add the new column/table/structure alongside the old one. Nothing that already depends on the old structure breaks, because the old structure still exists untouched.
  2. Migrate — backfill the new structure from the old one, and deploy application code that dual-writes: every write goes to both the old and new structure, keeping them in sync while reads still come from the old structure (or, once verified, from the new one).
  3. Contract — once every code path reads and writes only the new structure, and a monitoring window confirms nothing is still touching the old one, remove the old structure.

The critical property is that each of the three phases is its own deploy, separated in time from the others, and the system is in a fully valid, fully rollback-able state after every single one. There is no moment where "half the fleet expects the new schema and half expects the old" causes an error — both schema shapes coexist on purpose during the migrate phase.

Tradeoffs

Approach Benefit Cost
In-place migration (ALTER TABLE ... RENAME/DROP at deploy time) One step, simplest to write Requires the schema change and the code deploy to land atomically; any rolling deploy or migration lag causes requests to hit the wrong shape — real downtime or errors
Expand-contract Every intermediate state is valid; each phase is independently deployable and rollback-able; supports rolling deploys and canaries cleanly 3+ deploys instead of 1; dual-write phase adds real complexity (two writes must both succeed or the data drifts) and must be actively monitored, not left running indefinitely
Maintenance-window migration (take the app offline, migrate, bring it back up) Simplest to reason about, no dual-write complexity Straightforward downtime — unacceptable for most production systems with real users or SLAs

Expand-contract is not free: the dual-write phase is a genuine liability if left running too long. If dual writes silently fail on one side (e.g. the write to the new column succeeds but the write to the old column times out), the two structures drift apart, and the contract phase then deletes data nobody backfilled correctly. The pattern trades one big risky change for several small ones, but only pays off if each small change is verified before moving to the next.

When to use / when not to

  • Use for any schema or storage-format change touching a table or field that's actively read/written by a live system with a rolling or blue/green deploy model — renames, type changes, splitting one table into two, moving a field to a different service's database.
  • Especially necessary when the migration and the corresponding code deploy can't be guaranteed to land as a single atomic unit — which is true of almost every real deployment pipeline (canary releases, multi-region rollout, mobile clients that update on their own schedule).
  • Skip it for a system that can genuinely tolerate a short maintenance window (internal batch tooling, a system with a contractual maintenance window already) — the extra deploys and dual-write complexity aren't worth it if downtime is actually acceptable.
  • Skip it for purely additive changes (adding a new nullable column nobody reads yet) — that's already non-breaking on its own and doesn't need the full three-phase treatment, just the expand step.

Common pitfall

Treating the dual-write "migrate" phase as a one-time backfill script instead of an ongoing invariant that must be actively enforced and monitored. Teams write the backfill, verify counts once, and move straight to contract — but new writes that arrive between the backfill and the contract step, or app instances still running old code that only writes the old column, leave the new structure incomplete. The fix is to keep both writes active and add reconciliation checks (row counts, checksums) that must pass continuously for a defined bake period — not just once — before the old structure is ever dropped in the contract phase. Mobile/client-driven systems make this worse: an old app version can keep writing only the old shape for weeks after a backend deploy, well past when a team assumes migration is "done."

Engineering Lens

The pattern is the schema-change instance of a more general deployment principle: never make a single deploy carry two independent risks (a data-shape change and a code-behavior change) that must succeed together. Splitting them into separately-deployable, separately-verifiable steps is the same instinct behind feature flags and canary releases — reduce the blast radius of any one change and make every intermediate state safe to pause on or roll back from. The real engineering skill isn't writing the ALTER TABLE — it's defining what "safe to proceed to contract" means (what gets monitored, for how long, with what rollback trigger) before the expand phase ever ships.

Sources

Hermes Wiki