Developer/CostManagement/ReservedVsOnDemand/Fundamentals/spot-instances-and-interruption-handling

Spot Instances and Interruption Handling

Concept

Cloud providers sell spare, unused compute capacity at a steep discount (commonly 60-90% off on-demand pricing) with one condition: the provider can reclaim that capacity back with short notice — typically a two-minute warning on AWS EC2 Spot — whenever it's needed for on-demand or reserved customers instead. That reclaim is an interruption, not a failure in the traditional sense; the instance isn't crashing, it's being deliberately terminated because the discount comes with a standing risk premium the workload has to be architected to absorb. The deal only makes economic sense if the workload's architecture treats interruption as a routine, expected event rather than an exceptional one — which means the compute layer needs to be stateless (or checkpoint state externally), horizontally distributed across many instances so no single interruption takes down the whole job, and diversified across instance types/Availability Zones so a capacity crunch in one pool doesn't interrupt the entire fleet simultaneously.

The interruption-handling mechanics matter as much as the pricing: cloud providers send a termination notice (via the instance metadata service on AWS) some fixed window before reclaiming the instance, and a well-built workload listens for that signal and reacts — draining in-flight work, checkpointing progress, deregistering from a load balancer — rather than getting killed mid-task and losing the work. Auto Scaling groups configured for Spot will also request replacement capacity automatically, so a spot fleet self-heals its capacity even as individual instances come and go, provided the workload doesn't depend on any single instance's identity or local state surviving.

Tradeoffs

Approach	Cost	Interruption risk	Architectural requirement
On-Demand	Highest (baseline)	None	None — works with any architecture, including stateful singletons
Reserved / Savings Plans	Lower (committed-use discount)	None	Requires accurate long-term capacity forecasting to avoid paying for unused commitment
Spot	Lowest (60-90% off)	High — reclaimable with ~2 min notice at any time	Workload must be stateless/checkpointable, horizontally distributed, and tolerant of losing any individual instance

The tradeoff isn't just "cheaper but riskier" in the abstract — it's a direct tradeoff between cost savings and the engineering investment required to make interruption a non-event. A stateless batch worker that checkpoints every few seconds can run almost entirely on Spot with near-zero practical downside; a stateful singleton service with in-memory session state has no safe way to run on Spot at all without a substantial re-architecture.

When to use / when not to

Use Spot for stateless, horizontally scaled, interruption-tolerant workloads — batch processing, CI/CD build fleets, big-data/Hadoop-style jobs, and fault-tolerant web-tier fleets sitting behind a load balancer with enough headroom to absorb an instance dropping out.
Diversify across multiple instance types and Availability Zones in a Spot fleet — relying on a single instance type in a single AZ means a capacity crunch there interrupts the whole fleet at once instead of a fraction of it.
Don't run stateful singletons, databases holding the only copy of data, or anything where losing an instance mid-task loses unrecoverable work on Spot — the discount isn't worth the operational risk there.
A common production pattern is mixing pools — a baseline of On-Demand/Reserved capacity sized to the workload's non-negotiable floor, with Spot absorbing burst/elastic capacity on top, so an interruption degrades capacity rather than causing an outage.

Common pitfall

Adopting Spot for cost savings without actually building the interruption-handling logic — listening for the termination notice and draining gracefully — and instead discovering the gap only when a real capacity crunch interrupts a meaningful fraction of the fleet simultaneously and in-flight work is silently lost. The pricing discount is real and immediate; the architectural requirement it depends on is easy to skip in a proof-of-concept and only becomes visible as a gap once interruptions actually happen at scale in production.

Engineering Lens

Spot is a clean example of a cost pillar decision that's inseparable from a resilience pillar decision — the discount is real, but claiming it responsibly requires the same statelessness and graceful-degradation architecture that resilience engineering asks for anyway, which is exactly the kind of cross-pillar framing that reads as Principal-level thinking in a review. The sharper question to bring to a design discussion isn't "should we use Spot to save money" — it's "which parts of this system are already interruption-tolerant by design, and can capture Spot pricing essentially for free, versus which parts would need real re-architecture to tolerate it, where the savings might not justify the engineering cost."