Architecture/Fundamentals/infrastructure-as-code

Infrastructure as Code (IaC)

Concept

Infrastructure managed by hand — clicking through a cloud console, running one-off CLI commands, SSH-ing in to patch a config — accumulates drift almost immediately. Two environments that started identical diverge the first time someone fixes an incident directly in production and forgets (or doesn't bother) to replicate the change in staging. Nobody can answer "what does our infrastructure actually look like right now" with confidence, because the true state lives scattered across consoles and shell history rather than in any single reviewable place. Disaster recovery becomes a scramble to remember what was manually configured rather than a rebuild from a known definition.

Infrastructure as Code treats infrastructure the same way application code is treated: definitions live in version-controlled files, changes go through the same review process as any other code change, and the actual running infrastructure is produced by applying those files rather than by hand. Tools split along two models. Declarative tools (Terraform, AWS CloudFormation, Pulumi in its typical usage) have the author describe the desired end state — "this VPC, these three subnets, this security group" — and the tool computes and executes whatever diff is needed to get current infrastructure to match. Imperative tools (a bespoke shell/Ansible script) have the author describe the steps to take, in order, requiring the author to reason about current state, prior runs, and idempotency themselves.

The declarative model dominates modern infrastructure tooling because it shifts the hard problem — "what needs to change to reach this state" — onto the tool instead of the author, and because the desired-state file itself becomes a readable, diffable source of truth. Most declarative tools maintain a state file — a record of what they last created and its properties — that they diff the desired configuration against on every run; that state file becoming out of sync with reality (from manual out-of-band changes) is the single most common source of IaC pain in practice.

Tradeoffs

Approach	Source of truth	Drift risk	Review-ability	Rebuild speed
Manual (console/CLI/SSH)	Whatever's currently running — undocumented	High — every manual fix is a potential divergence	None — no diff, no review	Slow, error-prone; depends on institutional memory
Imperative scripts (Ansible, shell)	The script, but correctness depends on the author handling idempotency	Moderate — reruns can be unsafe if not carefully written	Some — script is reviewable, but effect on state isn't always obvious from reading it	Faster than manual, still sequential and script-order-dependent
Declarative IaC (Terraform, CloudFormation, Pulumi)	The desired-state file, checked against a state file	Low when strictly enforced (no manual changes allowed); state-file drift if that discipline slips	High — a `plan`/diff step shows exactly what will change before it happens	Fast — apply the same files to a fresh environment and get the same result

The trade isn't really about tooling preference — it's about whether infrastructure has a reviewable source of truth at all. Manual management is the fastest path for a single one-off change and the slowest path to recovering from an incident or standing up a second environment. Declarative IaC costs more upfront (learning the tool, writing the modules, enforcing "no manual changes" discipline) in exchange for infrastructure that's auditable, reproducible, and safe to change via the same pull-request review process as application code.

When to use / when not to

Use for any infrastructure expected to be recreated, scaled, or replicated across environments (dev/staging/prod) — the reproducibility payoff compounds every time the environment is touched again.
Use whenever infrastructure changes need the same review, audit trail, and rollback capability as application code — which is effectively always in a regulated or production-critical environment.
Use as the backbone for disaster recovery — being able to apply a known-good definition into a fresh region or account is a categorically faster recovery path than manual reconstruction.
Skip, or defer, heavy IaC tooling for genuinely disposable, single-use, throwaway resources (a scratch dev sandbox torn down within the hour) where the setup cost of formalizing it exceeds its lifetime value.
Don't treat IaC adoption as complete just because the files exist — if manual console changes are still allowed alongside it, the state file drifts and the "source of truth" claim becomes false without anyone noticing until an apply produces a surprising diff.

Common pitfall

Allowing manual changes to slip in alongside an IaC-managed environment "just this once" during an incident. The infrastructure now differs from what the state file believes is true, and the next routine plan either silently reverts the manual fix (surprising and possibly re-breaking something) or the team starts distrusting plan output entirely and stops reviewing diffs closely — which erodes the entire safety benefit IaC was adopted for. The discipline that actually makes IaC valuable — no out-of-band changes, full stop — is a process commitment, not a tooling feature, and it's the part organizations most often skip under incident pressure.

Principal Engineer Lens

IaC is a good lens for a broader Principal-level habit: treating "can I reproduce this system's state from a known definition" as a first-class design requirement, not an afterthought bolted on after the system already exists. The same question applies past infrastructure — can a database's current schema be derived from migration files, can a service's configuration be reconstructed from its deploy manifest, can an org's access grants be derived from a policy-as-code definition rather than a spreadsheet of one-off approvals. In an architecture review, "how would we rebuild this from scratch, and how do we know the definition we'd rebuild from actually matches production" is a sharper diagnostic than asking whether IaC tooling is in use at all — a team can have Terraform files that no longer describe reality just as easily as a team with no IaC at all.