Developer/Containers/Orchestration/Fundamentals/container-orchestration-the-reconciliation-loop

Container Orchestration: The Reconciliation Loop

Concept

Strip away the differences between Kubernetes, Nomad, and ECS and the same mechanism sits underneath all of them: a reconciliation loop. An operator declares what should exist — "3 replicas of this container, this image, this resource shape" — and a controller continuously compares that declared, desired state against the cluster's observed actual state, then issues whatever create/update/delete operations close the gap. It doesn't run once at deploy time and stop; it keeps running, forever, watching for drift and correcting it — a Pod dies, the controller notices the actual replica count fell below desired and creates a replacement, with no human or script involved.

The important design choice hiding in that description is that reconciliation is level-triggered, not edge-triggered. An edge-triggered system reacts to individual events — "a Pod died, so restart it" — and if that specific event is dropped (a network partition, a controller restart mid-processing), the system never finds out and drifts out of sync permanently. A level-triggered system instead recomputes the diff between desired and actual state from scratch on every reconciliation pass, regardless of what triggered it — so a missed event doesn't matter, because the next pass (there's always a next pass) will see the same discrepancy and correct it anyway. This is what makes orchestrators self-healing rather than merely "has a restart policy": the controller has no memory of what happened, only an ongoing comparison of two states, which makes it robust to controller crashes, missed events, and out-of-order updates in a way an imperative "if X happens, do Y" script fundamentally isn't.

Tradeoffs

Model	How it reacts	Benefit	Cost
Imperative scripting (run this container, restart-on-fail via a wrapper script)	Executes a fixed sequence once; retries are hand-coded	Simple to read top to bottom; no separate control-plane component needed	Doesn't self-heal after the script exits; state drifts silently as manual changes accumulate outside the script's knowledge
Edge-triggered event handling	Reacts to individual state-change events as they arrive	Fast, low-overhead reaction to changes	A dropped or duplicate event permanently desyncs actual from desired unless paired with periodic resync
Level-triggered reconciliation (Kubernetes/Nomad controllers)	Recomputes desired-vs-actual diff on every pass, independent of trigger	Self-healing; robust to missed events, controller restarts, out-of-order updates	Convergence isn't instant — "apply" means "eventually true," not "true right now," which changes how you reason about and debug the system

The practical cost of the level-triggered model shows up in debugging: reasoning about "why hasn't this converged yet" requires understanding controller reconcile intervals and dependency ordering, which is a different mental model than reading a linear script top to bottom.

When to use / when not to

Reach for a reconciling orchestrator once there's more than a handful of container instances that need coordinated scheduling, health-checking, and restart behavior — past the point where a human or a simple supervisor script can track actual state by hand.
A managed, lighter-weight option (AWS ECS/Fargate) gives the same reconciliation model with far less operational surface than self-hosting a full Kubernetes control plane — worth ruling out before adopting Kubernetes specifically.
Don't adopt a full orchestrator to manage a single service on a single host — a process supervisor (systemd, a container runtime's own restart policy) already gives basic reconciliation at that scale without the added control-plane complexity.
Don't fight the model by editing actual state directly and expecting it to stick (e.g., manually changing a running Pod count instead of the Deployment's declared replica count) — the next reconciliation pass will silently revert it, because from the controller's view that edit was drift, not an intent change.

Common pitfall

Treating a change to desired state like a synchronous imperative command — expecting the cluster to reflect it immediately — instead of watching for convergence. This shows up most often as debugging a rollout that "isn't working" when it's actually still converging (a node lacks capacity, an image pull is slow, a readiness probe hasn't passed yet), or as directly patching the live/actual state to fix a symptom, which the next reconciliation pass quietly undoes because the controller only ever answers to the declared desired state, never to a manual edit made around it.

Engineering Lens

The reconciliation loop is worth understanding on its own, independent of any specific orchestrator's API, because it's the same pattern that shows up anywhere a system needs to stay correct despite partial failures and missed messages — a config-sync agent, a DNS record manager, an infrastructure-as-code apply loop. The transferable insight for a design review is naming why level-triggered reconciliation is more robust than "retry on failure": it doesn't need to know what failed, only what's currently true versus what should be true, which is a strictly weaker (and therefore more failure-tolerant) requirement than tracking every individual event correctly.