Hermes Wiki

Blue-Green and Canary Deployments

Concept

Both patterns exist to answer the same question — how do you ship a new version without an outage or an unrecoverable bad release — but they attack it differently:

  • Blue-Green — two identical, full-size production environments exist at once ("blue" = current, "green" = new). Traffic cuts over from blue to green all at once (usually at a load balancer or DNS layer) once green is verified healthy. Rollback is just cutting traffic back to blue, which is still fully running.
  • Canary — the new version is released to a small slice of production traffic (often 1-5%), monitored against key metrics, then gradually ramped up (5% → 25% → 100%) only if those metrics stay healthy at each step. No second full environment is required — just the ability to route a fraction of traffic to the new version alongside the old.

Both eliminate the "big-bang" deploy where 100% of traffic hits untested-in-production code simultaneously, but they trade off differently on speed, cost, and blast-radius control.

Tradeoffs

Dimension Blue-Green Canary
Infrastructure cost Double capacity during rollout Minimal — new version rides alongside old
Rollback speed Instant (flip traffic back) Fast but not instant (drain canary traffic)
Blast radius on a bad release Briefly 100% until detected Capped by canary percentage the whole time
Detection requirement Health check before cutover Continuous metric comparison during ramp-up
Database/schema changes Hard — both environments often share one DB, so schema must be backward-compatible either way Same constraint applies

The core tradeoff: blue-green optimizes for fast, clean rollback at the cost of running double infrastructure temporarily; canary optimizes for small blast radius from the start at the cost of a slower rollout and the need for real per-cohort metrics, not just a single health check.

When to use / when not to

  • Blue-Green fits releases where a clean instant rollback matters more than gradual exposure — infrequent, high-confidence releases, or environments where running two full stacks briefly is cheap relative to downtime risk.
  • Canary fits high-traffic services where a subtle regression (a memory leak, a slow query path) might not show up in a health check but would show up in error-rate or latency metrics after real traffic hits it for a while.
  • Combine them in mature setups: canary a small percentage first to catch the class of bug a health check misses, then blue-green-style full cutover once the canary looks clean.
  • Neither is worth the operational complexity for a low-traffic internal tool where a five-minute rollback via redeploy is an acceptable cost — matching the deployment strategy to the actual blast radius at stake avoids over-engineering the release pipeline.

Common pitfall

Assuming either pattern is a substitute for backward-compatible schema migrations. Both patterns run old and new code paths against the same database during the transition window (blue-green during health-check verification, canary throughout the whole ramp), so a schema change that isn't backward-compatible breaks the old version the moment it's deployed — well before the rollout even finishes. The database migration has to be its own multi-step, backward-compatible process (expand/contract), independent of whichever traffic-shifting pattern is used on top of it.

Engineering Lens

The distinction that reads as Principal-level in a review isn't "which one is better" — it's correctly matching blast-radius tolerance to release frequency and traffic volume, and being explicit about the failure mode each pattern still doesn't solve (schema compatibility, for either). Being the person who flags "our canary metrics window is too short to catch a slow memory leak" before a release, rather than after an incident, is exactly the kind of judgment that separates someone who's read about deployment patterns from someone who's operated them under real traffic.

Sources

Hermes Wiki