Architecture/Fundamentals/slo-sli-error-budgets

SLIs, SLOs, and Error Budgets

Concept

Google's SRE framework for turning "is the system reliable?" from an opinion into a governed, quantitative decision, built from three layered pieces:

SLI (Service Level Indicator) — a directly measured metric of user-facing behavior, expressed as a ratio: good events / valid events × 100. For example, the proportion of HTTP requests to a checkout service that do not return a 5xx status, or the proportion of successful responses completing within 500ms. An SLI is chosen per critical user journey (checkout, login, search), not per infrastructure component — CPU usage isn't an SLI, "percentage of checkouts that succeeded" is.
SLO (Service Level Objective) — a target value for an SLI over a compliance period, e.g., "99.9% of checkout requests succeed, measured over a rolling 30 days." The target should be set from historical baselines and user tolerance, not aspiration — if users can't tell the difference between 300ms and 500ms latency, the SLO threshold belongs at the higher, cheaper-to-hit value.
Error budget — the inverse of the SLO: 100% − SLO% of allowed unreliability over the compliance window. A 99.9% availability SLO over 30 days yields roughly 43 minutes of allowed downtime; a 99.99% SLO yields about 4 minutes. The budget is a spendable allowance, not a penalty — it exists precisely so teams have an explicit, pre-agreed amount of risk they're allowed to take.

SLA (Service Level Agreement) is a related but distinct fourth term worth naming for contrast: it's the externally-facing, usually contractual promise (with financial or legal consequences for breach) that SLOs are commonly set stricter than, so internal alerting fires and gives time to react before the customer-facing SLA is actually violated.

The mechanism that makes this a governance tool rather than just a dashboard: when the error budget is healthy, product/dev teams can ship fast, take risks, run experiments. When the budget is nearly exhausted, that same team is incentivized to slow down, freeze risky releases, and prioritize reliability work — without SRE having to police it externally. The team becomes self-regulating because the budget, not a person, is the constraint.

Tradeoffs

Choice	What you gain	What it costs
Setting SLO very high (99.99%+)	Tight error budget forces rigorous engineering discipline, strong user trust for critical paths	Exponentially more expensive per additional nine; shrinks the budget available for velocity/experimentation; may be wasted precision if users can't perceive the difference
Setting SLO loose (99.0-99.5%)	Large error budget, more room for fast iteration and risk-taking	Under-promises reliability for services where a single bad journey (payment, auth) has outsized user/business impact
Rolling window (30-day) vs. calendar window	Smooths noise, matches how users actually perceive ongoing reliability	Slower to reflect a fixed one-time incident's long-run effect; harder to reason about "did this month meet target" for reporting
Per-user-journey SLOs vs. one system-wide SLO	Precisely targets what actually matters to users (checkout vs. background job)	More SLOs to define, measure, and maintain; requires genuine agreement on which journeys are "critical"
Burn-rate alerting (e.g., alert at 2x baseline burn) vs. static threshold alerting	Catches budget-consuming incidents early, before the whole budget is gone	Requires computing burn rate correctly (short window for fast burns, long window for slow ones) — a naively-tuned burn-rate alert either pages too often or misses slow leaks

The central tradeoff underneath all of these: 100% reliability is the wrong target for almost any system. The cost of chasing additional nines grows non-linearly while the marginal user benefit flattens quickly — the error budget exists to make that tradeoff explicit and negotiated up front, instead of an implicit, unexamined assumption that "more reliable is always better."

When to use / when not to

Use SLOs and error budgets wherever a service has distinguishable, measurable user journeys and a team that can act on the signal (slow down releases, invest in reliability work) when the budget burns down. This is the common case for any production service with real users and an ongoing release cadence.
Skip formal SLOs for internal tools, prototypes, or one-off batch jobs with no meaningful "user journey" to define an SLI around — the overhead of designing SLI/SLO/error-budget machinery isn't worth it until there's a real reliability-vs-velocity tension to govern.
Don't set an SLO you have no mechanism to act on. An error budget only works as a governance tool if burning through it actually changes team behavior (release freezes, reprioritized on-call work) — a target nobody enforces is just a number on a dashboard.
Don't copy someone else's SLO number. A payments-authorization SLO and a "recommend articles" SLO should not carry the same target — the target should reflect what breaks for the business and the user when that specific journey fails, not a generic industry number.

Common pitfall

Picking an SLI that doesn't reflect what users actually experience, most commonly by measuring server-side health (CPU, uptime of a process) instead of the user-facing journey (did the checkout request succeed, was the page usably fast). A service can be "up" by infrastructure metrics while every real checkout attempt is failing due to a downstream dependency — the SLI needs to be defined at the boundary the user actually crosses, not at whatever's easiest to instrument. A second common pitfall: treating the error budget as a penalty to hide from instead of a decision tool — teams that don't have a pre-agreed error budget policy (what actually happens when the budget is nearly gone — freeze non-critical releases? mandatory postmortem? reprioritize the sprint?) end up either ignoring budget exhaustion entirely or litigating the response after the fact, both of which defeat the point of having a budget at all.

Principal Engineer Lens

The SLO/error-budget conversation is one of the clearest places a Principal Engineer earns credibility in an architecture review, because it forces a business tradeoff into the open instead of leaving "how reliable should this be" as an implicit assumption everyone quietly disagrees about. The skill isn't reciting the formula — it's picking the right critical user journey and the right target for this specific system's failure cost. A trading order-entry system and a research-notebook internal tool do not deserve the same reliability target, and defending that difference with a concrete cost/benefit argument (cost of the next nine vs. cost of a bad trade or a stalled settlement) is exactly the kind of judgment that separates "we monitor uptime" from "we've decided, deliberately, how much unreliability this system is allowed and why." In Fintech and Capital Markets specifically, this framing does real work: a payments-authorization or trade-execution path often needs a strict, tightly-budgeted SLO (minutes of allowed downtime per month, if that) precisely because the downstream cost of an outage is asymmetric — a failed trade or a stuck settlement isn't just user-annoying, it's a regulatory and financial exposure. Meanwhile a supporting reporting or analytics service in the same platform can run a much looser SLO, and defending why those two systems get different targets, backed by the actual business cost of each failure mode, is the argument a Principal Engineer should be able to make crisply in a review.

Reel Script

Setup: A VP asks "is the checkout service reliable enough?" Nobody in the room can answer with a number — everyone has an opinion, nobody has a target, and there's no way to know whether last month's three outages were "fine" or "way too many."

Concept walkthrough: Walk through the three layers in order: first pick the SLI — a measurable ratio tied to a real user journey, like the percentage of checkout requests that don't 5xx. Then set the SLO — a target for that SLI over a window, like 99.9% over a rolling 30 days. Then derive the error budget automatically: 100% minus 99.9% is 0.1%, which works out to roughly 43 minutes of allowed bad-checkout time per month. That 43 minutes is now a real, spendable, trackable number instead of a vague reliability aspiration.

Real example tie-in: Take Google Cloud's own worked example: an e-commerce checkout journey with a 99.9% availability SLO and a 500ms latency SLO, tracked over a rolling 30-day window, with dashboards showing real-time error-budget burn and alerts firing at something like 2x the baseline burn rate — fast enough to catch a bad incident well before the whole month's budget is gone, instead of finding out at the end of the month that the target was blown.

Tradeoffs & alternatives: Contrast the extremes: chase 99.99% and you get a tiny error budget, which forces extreme engineering discipline and slows down risky releases — worth it for a payments-authorization path, probably wasteful for an internal admin tool. Set the target too loose and you protect velocity but silently accept an unreliability level nobody actually agreed was okay. The real craft is picking the target per user journey, not applying one number system-wide, and pairing it with a pre-agreed error budget policy — what actually happens operationally when the budget runs low — so the number changes behavior rather than just sitting on a dashboard.

Principal Engineer takeaway: The measure of whether this is being done well isn't "do we have an SLO dashboard" — it's whether burning the error budget actually changes what the team does next: a release freeze, a reprioritized sprint, a mandatory reliability sprint. An SLO nobody acts on is decoration; an SLO that visibly shifts team behavior when it's at risk is the entire point of the framework.

Architecture Index
Three Pillars of Observability — observability (metrics/logs/traces) is the instrumentation layer that makes an SLI measurable in the first place; SLOs and error budgets are the governance layer built on top of that signal, turning "we can see what's happening" into "we've agreed on a target and what happens when we miss it."

Sources: