Hermes Wiki

Rollback vs. Roll-Forward

Concept

When a deployment goes bad, "just roll it back" sounds simple until the change touches a database, at which point there are two genuinely different recovery patterns with different guarantees. Rollback restores the system to its previous state as if the bad change never happened — for application code, that's redeploying the prior artifact or image; for a database schema, it means literally reversing the migration, taking the schema from v4 back to v3. Roll-forward instead acknowledges the bad state occurred and moves ahead to a new version — v5 — that happens to correct the problem and land the system back in the desired state, without ever pretending v4 didn't happen.

The distinction matters most for stateful systems, because application code and database state don't share the same recovery properties. Application rollback is comparatively clean: redeploy the previous artifact and the running code reverts completely. Database rollback is not — data written under the new schema doesn't disappear when the schema reverts, and certain operations (dropping a column, dropping a table) destroy information that a schema rollback alone cannot restore; that requires a real backup/restore, not just an "undo" migration script. This is why forward-only migration discipline exists: keeping every schema change additive and backward-compatible means the previous application version keeps working against the new schema, which turns "rollback" for the application layer into "redeploy the old binary" without ever needing to touch the database at all.

Tradeoffs

Approach Guarantees Risk Compliance/audit trail
True rollback (revert schema to prior version) System returns to exactly its prior known-good state Cannot safely undo destructive changes (dropped columns/tables) without a backup restore; data written under the new state may be lost Weaker — the deployment history shows a state that was later erased, not corrected
Roll-forward (new version that corrects the problem) Never loses data written under the "bad" state; audit trail stays intact Requires you to actually build and ship a fix quickly — not always fast under incident pressure Stronger — every state the system was actually in remains part of the recorded history, which matters for regulated environments
Backup/restore Only reliable way to recover data lost to a destructive schema change Slow, and loses any data written between the backup and the restore point N/A — a last resort, not a routine rollback mechanism

The general recommendation from database-migration tooling vendors (Redgate, Harness) is to prefer roll-forward for live production databases specifically because it preserves the audit trail and avoids the data-loss risk inherent in reversing a schema change — true rollback is reserved for cases where the bad change left the database in a genuinely dangerous state that can't wait for a forward fix.

When to use / when not to

  • Prefer roll-forward as the default for any live production database change — it's simpler to reason about, avoids destructive-rollback data loss, and keeps the audit trail intact for compliance purposes.
  • Reserve true schema rollback for cases where the bad state is actively dangerous (data corruption in progress, a security-relevant misconfiguration) and waiting for a forward fix isn't acceptable — and even then, be explicit that any data written under the bad state may be lost.
  • Application-code rollback (redeploy the previous artifact) stays simple and low-risk regardless of the database strategy chosen, as long as the schema changes underneath it were forward/backward-compatible — this is the entire reason expand/contract migration discipline exists.
  • Never treat a rollback script as a substitute for a real backup/restore strategy — a rollback script that "undoes" a dropped column can recreate the column, but not the data that was in it.

Common pitfall

Assuming rollback is symmetric with the original deploy — that because deploying forward was safe, reversing it is equally safe. It usually isn't, the moment a database is involved. Rolling back a migration that dropped a column doesn't restore the dropped data; rolling back a migration that renamed a table can break any code (including the previous application version now being restored) that already started expecting the new name. The asymmetry between "deploying a change" and "un-deploying a change" is exactly why forward-only, backward-compatible migrations are the safer default — they turn "rollback" into "redeploy old code against a schema that was designed to still support it," which sidesteps the asymmetry problem entirely instead of trying to solve it after the fact.

Engineering Lens

The Principal-level version of "we have a rollback plan" is being able to say, precisely, which parts of the system the plan actually restores and which parts it can't — application state reverts cleanly, but any data written to the database under the failed version either survives (roll-forward, or a backward-compatible rollback) or requires an explicit backup-restore path that's been tested, not assumed. The gap that actually causes incidents isn't "we don't have a rollback strategy" — it's discovering, mid-incident, that the rollback strategy only ever covered the application layer, and the database half was never defined or exercised outside of a real emergency.

  • Rolling Deployment
  • Automated Post-Deployment Verification (unresolved)
  • Golden Images and Immutable Infrastructure (unresolved)

Sources

Hermes Wiki