Design an Incident Response and On-Call Escalation System
Scenario prompt
Design the paging and escalation system that sits between "a monitoring system fired an alert" and "a human is actively working the problem," across an organization with multiple teams and on-call rotations. Requirements:
- Alerts arrive from many independent monitoring sources and must route to the correct team/service owner
- If the primary on-call doesn't acknowledge within N minutes, the page must escalate — to a secondary, then to a team lead — automatically
- The system must not fail silently if the paging provider itself has an outage
- It must reduce alert fatigue (noisy, low-value pages erode trust and get ignored) without ever silently dropping a genuinely high-severity page
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd design the escalation state machine and what you'd do if the paging provider itself goes down mid-incident.
Model solution
Ingestion — a single alerting gateway, not N monitoring tools each paging directly. Every alert source (metrics thresholds, log-based alerts, synthetic checks, error-rate SLO burn alerts per SLIs, SLOs, and Error Budgets) feeds into one aggregation layer that fingerprints each alert (service + condition + resource) for dedup and correlation, then routes it through an ownership table to the right team's on-call schedule. Without this layer, alert fatigue is structural — five monitoring tools each independently paging on the same underlying incident is indistinguishable, from the human's perspective, from five unrelated incidents.
Escalation as an explicit, durable state machine — not a fire-and-forget notification. Primary on-call is paged; a durable timer starts. If unacknowledged within N minutes, the policy escalates to secondary; if still unacknowledged, to the team lead or a broader on-call manager rotation. This has to be backed by a durable scheduled job (survives the paging service restarting), not an in-memory timer, or an escalation silently never fires because the process that was tracking it happened to restart.
Fail-safe on the paging provider itself failing — fail toward more people notified, not fewer. A single paging vendor going down can't be allowed to mean nobody gets paged. The standard mitigation is a multi-channel, multi-provider notification path (push + SMS + automated phone call, ideally through more than one vendor) plus a documented fallback: if the escalation policy itself can't be executed (the paging system is down), broadcast to a wider fallback channel (e.g., a cross-team incident channel) rather than failing closed. This mirrors the fail-open reasoning in Design a Rate Limiter — a control system's own outage should degrade toward safety, not toward silence.
Noise reduction without silent drops. Severity classification (tied to actual business/customer impact, not just a threshold crossing) determines urgency and escalation aggressiveness. Correlated alerts (a dependency outage triggering alerts in ten downstream services) collapse into a single incident rather than paging on-call ten separate times. Conditions that self-resolve auto-close the page. None of this ever means dropping a high-severity alert — it means grouping and prioritizing what already fires, with the grouping/suppression logic itself observable and auditable, so "why didn't this page me" is always answerable.
A full timeline as a first-class output, not an afterthought. Every page, acknowledgment, and escalation step gets logged with timestamps — this is the raw material for the postmortem process and for tuning the system itself (which alerts get ignored, which escalate too aggressively, whether on-call load is fairly distributed).
Gaps to revisit
- Alert correlation is the hardest part in practice — how do you reliably tell "one root cause, ten symptoms" from "ten unrelated incidents that happen to be concurrent," especially across services owned by different teams?
- On-call fairness: rotation scheduling across timezones, and preventing escalation policies from quietly overloading one person (e.g., a secondary who's also secondary on three other rotations)
- How does severity get defined and kept consistent across teams — a SEV1 for the payments team and a SEV1 for an internal tooling team shouldn't mean wildly different actual impact
- Should the escalation policy itself be a target for chaos engineering (per Chaos Engineering) — periodically verifying pages actually reach humans, not just that the code path exists?
Engineering Lens
The real design problem here isn't "how do I send a notification" — it's "how do I guarantee a human eventually takes action, even when the very system responsible for guaranteeing that is itself degraded." That's a resilience-of-the-resilience-system problem, structurally identical to designing a circuit breaker's own failure mode (Circuit Breaker Pattern) or a rate limiter's fail-open behavior — the pattern of "what does my safety mechanism do when it, itself, breaks" recurs constantly at Principal scope and is worth naming explicitly in any review rather than leaving implicit. It also connects directly to the observability pipeline design in this same folder: alerting is only as good as the telemetry feeding it, and a Principal Engineer should be able to reason about both halves as one system, not two.