Hermes Wiki
Developer/CostManagement/CloudCostOptimization/Challenges/design-a-cost-anomaly-detection-system-for-a-multi-account-cloud-estate

Design a Cost Anomaly Detection System for a Multi-Account Cloud Estate

Scenario prompt

You're the architect responsible for cost governance across a company's cloud estate that grew organically to 40+ AWS accounts (or GCP projects), owned by different teams, with no consistent tagging discipline. Requirements:

  • Detect an unexpected cost spike (a runaway Lambda in a retry loop, an accidentally public S3 bucket driving egress, an autoscaling group stuck scaling up) within hours, not at month-end billing review
  • Attribute spend accurately to the owning team/product even though tagging compliance is inconsistent across accounts
  • Avoid alert fatigue — normal day-to-day cost variance (a marketing campaign, a legitimate traffic spike) shouldn't page anyone
  • Support both automated response for known-bad patterns and human-in-the-loop review for ambiguous spikes

Mihir's attempt

[!todo] Write your own attempt here before reading the model solution below — how you'd detect a spike faster than month-end billing, and how you'd attribute cost when tagging is inconsistent.

Model solution

Centralize and normalize billing data before trying to reason about it. Point every linked account's detailed billing export (AWS Cost and Usage Report, or GCP's BigQuery billing export) at one central account/dataset, at the finest granularity available (hourly, per-resource where possible). Anomaly detection on daily-aggregated, account-level totals is too coarse to catch a single misbehaving Lambda before it's burned four figures — the granularity you detect at has to match the granularity a real incident happens at.

Use a rolling, seasonality-aware baseline instead of a static threshold. A flat "alert if daily spend > $X" rule either fires constantly during legitimate peaks (a marketing campaign, Black Friday) or stays silent during a real anomaly that happens during a naturally quiet period. A rolling baseline per service/account — comparing today's spend-by-hour against the same weekday's trailing average, with a tolerance band — catches genuine deviation from that resource's own normal pattern rather than from an arbitrary global number. This is the same instinct behind SLIs, SLOs, and Error Budgets: define "normal" from the system's own history, not a guessed constant.

Solve attribution with enforced tagging where you can, and a fallback heuristic where you can't. Tag policies and SCPs (or GCP Organization Policies) that block resource creation without a team/cost-center tag close the gap going forward, mirroring the centralized-least-privilege governance model in Pinterest: Resource Provisioner Pipeline. But untagged legacy resources still exist, so attribution needs a fallback: map by account ownership (if accounts are reasonably team-scoped) or by resource-creation IAM principal as a best-effort second signal, surfaced as "attributed" vs. "unattributed — needs owner" rather than silently guessing.

Tier alerts by severity and route to the owning team, not a central SRE inbox. A spike of $50/day on a $2,000/day baseline is noise; a spike of $50/day on a $10/day baseline is a real signal even though the absolute dollar amount is small — percentage deviation relative to baseline, not absolute dollars, is what should drive severity. Route confirmed anomalies straight to the owning team (via the attribution above) with the specific resource implicated, and reserve a central cost-governance channel for cross-account patterns or anything unattributed. For a small set of well-understood bad patterns (an idle dev/test instance left running over a weekend, a bucket with public-read egress), auto-remediate directly rather than paging a human — the same "known-bad-pattern gets automated, ambiguous case gets a human" split used in Chaos Engineering blast-radius containment.

Gaps to revisit

  • Detection latency is bounded by how quickly the cloud provider's billing data itself lands (AWS CUR can lag 24+ hours) — what's the fastest true "near-real-time" signal available (e.g., CloudWatch metrics as a proxy for spend before billing data catches up), and is it worth building a second, noisier fast path alongside the accurate slow path?
  • Multi-cloud estates multiply this problem — does a unified anomaly model even make sense across providers with different billing granularity and export formats, or does each cloud need its own detector feeding a shared alerting/attribution layer?
  • False-positive tuning is an ongoing tax — how do you measure and report "alert precision" over time so the system earns trust instead of getting muted by the teams it's supposed to warn?

Engineering Lens

The tension underneath this system is the same one underneath most cost-governance work: perfect real-time detection and perfect attribution both cost real engineering effort, and neither is free relative to the dollars they save. Naming the ROI explicitly — "this catches spikes within 2 hours instead of 30 days, and the historical incidents we'd have caught total $X" — is what turns a cost-anomaly system from a nice-to-have into something finance will fund, and is exactly the kind of tradeoff-with-numbers argument a Principal Engineer needs to make in a budget review rather than an architecture review. It also generalizes past cost: any detection system built on a noisy, high-cardinality signal (cost, security events, latency) faces the same baseline-vs-threshold and precision-vs-recall tradeoffs, so getting fluent with this pattern here pays off well outside FinOps.

Hermes Wiki