Hermes Wiki
Synthesis/hands-on-slo-build-instantiates-three-abstract-architecture-notes-at-once

Synthesis: The Hands-On SLI/SLO Build Is a Live Instance of Three Abstract Architecture Notes at Once

The connection

SLI/SLO Observability, Built By Hand Before Any Vendor Touches It — new today — is a hands-on FastAPI/OTel/Prometheus/Sloth/Alertmanager/Grafana build that, without citing any of them, is a concrete worked example of three Architecture notes that have sat unconnected to any real implementation since they were written:

  • SLIs, SLOs, and Error Budgets (2026-07-23) defines burn rate abstractly as "how fast the error budget is being consumed relative to a sustainable pace" and calls out burn-rate alerting's central tuning problem: "a naively-tuned burn-rate alert either pages too often or misses slow leaks." The blog post supplies the concrete arithmetic the fundamentals note only gestures at — (1 − 0.95) / (1 − 0.99) = 5x — and a live dashboard showing that exact 5-6x number, plus Sloth's multi-window burn-rate rules as the actual implementation of the fundamentals note's "burn-rate alerting" row.
  • Design an Observability Pipeline for a Microservices Platform (2026-07-28) prescribes "ship a thin shared library... auto-instruments HTTP/gRPC calls... OpenTelemetry is the pragmatic default" as the model solution to standardizing instrumentation across teams. The blog post is that exact prescription built and run: FastAPI + OTel SDK auto-instrumenting HTTP, a Collector re-exporting to Prometheus, one shared correlation path from app to dashboard.
  • Design an Incident Response and On-Call Escalation System (2026-08-04) names group_wait/group_interval/durable-timer semantics abstractly ("a durable timer starts... backed by a durable scheduled job") as the mechanism that makes escalation reliable. The blog post's own glossary defines Alertmanager's actual three timers (group_wait, group_interval, repeat_interval) and documents hitting the exact failure mode the challenge note warns about in the abstract: re-running load against an already-firing alert produced zero new emails, which "looked like a bug but was repeat_interval working exactly as designed."

Why this wasn't visible before

All three Architecture notes are speaking in the register of "here's what a well-designed system does," written as guidance for a future design review — none of them are grounded against a system that was actually built, run, and debugged. The blog post is dated today and sits in BlogPosts/, a folder this vault's cross-linking has historically treated as separate from Architecture/'s system-design layer (different audience, different register — teaching content vs. reference notes). Nothing routes a reader from one to the other by default.

What this suggests

  • This is the rare case where a "Real bugs I hit building this" section in a blog post can retroactively validate an abstract note's own "Common pitfall" and "Gaps to revisit" sections — the coarse-histogram-bucket bug and the good/bad-inversion bug are concrete instances of the general problem slo-sli-error-budgets.md's pitfall section names ("picking an SLI that doesn't reflect what users actually experience"), just one layer more mechanical (a bucket boundary, not a bad metric choice).
  • Worth adding a Related link from all three Architecture notes back to this blog post the next time any of them are revisited — it's the first artifact in the vault that closes the loop from "here's the theory" to "here's what happens when you actually run it," which the observability-pipeline challenge's own gap list ("What happens if the observability pipeline itself is degraded by the same incident it's supposed to help diagnose") doesn't yet have a real example for, but the blog post's Alertmanager container-state gotcha (docker compose up -d --force-recreate, notification log lives in the writable layer) is a small instance of exactly that class of problem.
  • The blog post's own "If this were Dynatrace/Datadog" comparison tables already do the vendor cross-referencing that would otherwise be this note's job for Dynatrace, DataDog, AWS CloudWatch, and AWS Managed Prometheus — no need to duplicate that here, but it means the blog post is simultaneously the concrete instance of the Architecture notes above and a de facto comparison note for the Tools/ observability cluster, worth remembering as a single high-leverage artifact next time either area gets revisited.
Hermes Wiki