Golden Image and Immutable Infrastructure
Concept
A golden image is a pre-built, validated, versioned artifact — a VM image, container image, or device firmware image — that bakes in the OS, dependencies, and application config as a single unit. Immutable infrastructure is the deployment philosophy this artifact enables: instead of logging into a running server and changing it in place (installing a package, editing a config file), you build a new image with the desired state already baked in, deploy fresh instances from it, and destroy the old ones. Nothing running in production is ever modified after it starts.
This is a direct inversion of the older configuration-management model (Chef, Puppet, Ansible in their classic "converge a live server toward desired state" mode), which manages drift — the gradual divergence of a server's actual state from its intended state as ad hoc changes accumulate — by periodically re-applying configuration to correct it. Immutable infrastructure doesn't manage drift, it eliminates the condition that produces it: a server that's never modified after creation cannot drift, because there's no "after" state to diverge in. HashiCorp's Packer is the standard tool for building these images from code — a single declarative template can produce an AWS AMI, a GCP Compute Image, and an Azure VHD from the same source definition, and a hosted registry (HCP Packer) tracks each image's version, ancestry, and revocation status.
Tradeoffs
| Approach | Benefit | Cost |
|---|---|---|
| Immutable (golden image, replace-don't-patch) | Deployments are atomic (succeed completely or don't happen); drift is structurally impossible; rollback is "redeploy the previous image," a single well-tested operation | Requires all state that must survive a redeploy to live outside the instance (external DB, object storage) — an instance can't safely hold data on local disk if it might be destroyed and replaced; image build ("bake") time adds latency to every change, even a one-line config tweak |
| Mutable (Chef/Puppet/Ansible, converge live servers) | Works with existing systems incrementally, no re-bake required for small changes, teams already familiar with the tooling and the running systems | Drift accumulates over time as ad hoc changes and manual fixes pile up outside the tracked configuration; "redeploy the previous version" isn't well-defined unless every historical state was itself captured as a reproducible artifact — which mutable infrastructure doesn't inherently do |
| Hybrid (immutable images + light in-place config for fast-changing values like feature flags) | Gets atomicity/rollback for the bulk of the system while avoiding a full rebuild for values that change often | Splits "what's baked into the image" from "what's injected at runtime" — a design decision that has to be made deliberately per value, not left implicit |
The practical cost that gets underestimated: immutable infrastructure only delivers fast rollback if every golden image is retained and versioned, not just the latest. "Redeploy the previous image" is fast and safe when that image is sitting in a registry ready to go; it becomes a slow, improvised rebuild under incident pressure if the previous version was never kept as a discrete, addressable artifact in the first place.
When to use / when not to
- Use for any repeatable deployment target where consistency and rollback-ability matter — VM/container fleets, and especially device/firmware upgrade tooling, where "what exactly is running right now" needs to be a single answerable fact, not an accumulated history of live edits.
- Version and retain every golden image, not just the current one, and tag each with what it contains — the point of immutability is lost if only the newest image is kept, since there's then nothing concrete to roll back to.
- Less suited to state that genuinely needs to live on the instance itself and can't be externalized cheaply (some legacy stateful workloads) — immutable infrastructure works best when persistent state is already, or can be made, external to the compute layer.
- Don't treat "immutable" as meaning "never changes" — it means the running instance never changes; the desired state changes constantly, it just changes by building and deploying a new image, not by editing a live one.
Common pitfall
Treating image-based deploys as automatically giving you fast rollback, without actually keeping old images available. The conceptual simplicity — "just deploy the previous image again" — assumes that previous image was versioned and readily retrievable. If the pipeline only retains the latest golden image (to save storage, or because retention was never designed in), "roll back to the previous version" during an incident means re-locating or re-building that version under time pressure, which is exactly the slow, error-prone process immutability was supposed to eliminate. Golden-image discipline requires designing retention and addressability from the start, not just adopting the image-based build step.
Engineering Lens
The design-review question immutable infrastructure answers isn't "do we use images" — most teams do somewhere — it's whether the pipeline can name, right now, exactly which artifact is running in production and redeploy the one before it as a single tested operation. A team that bakes images but only keeps the latest one has the build discipline of immutable infrastructure without the rollback benefit, which is usually the whole reason the pattern was adopted in the first place. The stronger review answer is naming the actual retention policy (how many versions kept, for how long, how a specific one gets redeployed) rather than "we use Packer."
Related
- Post-Implementation Verification (PIV) — a golden-image rollback is only safe to trigger automatically if PIV can reliably detect that the new image needs rolling back
- Recreate/Big-Bang — whether a golden-image push is staged or pushed everywhere at once is an orthogonal decision from whether the artifact itself is immutable