Architecture/Challenges/design-a-cost-anomaly-detection-system

Design a Cost Anomaly Detection System for a Multi-Account Cloud Estate

Scenario prompt

You're the architect responsible for cost governance across a company's cloud estate that grew organically to 40+ AWS accounts (or GCP projects), owned by different teams, with no consistent tagging discipline. Requirements:

Detect an unexpected cost spike (a runaway Lambda in a retry loop, an accidentally public S3 bucket driving egress, an autoscaling group stuck scaling up) within hours, not at month-end billing review
Attribute spend accurately to the owning team/product even though tagging compliance is inconsistent across accounts
Avoid alert fatigue — normal day-to-day cost variance (a marketing campaign, a legitimate traffic spike) shouldn't page anyone
Support both automated response for known-bad patterns and human-in-the-loop review for ambiguous spikes

Mihir's attempt

[!todo] Write your own attempt here before reading the model solution below — how you'd detect a spike faster than month-end billing, and how you'd attribute cost when tagging is inconsistent.

Model solution

Centralize and normalize billing data before trying to reason about it. Point every linked account's detailed billing export (AWS Cost and Usage Report, or GCP's BigQuery billing export) at one central account/dataset, at the finest granularity available (hourly, per-resource where possible). Anomaly detection on daily-aggregated, account-level totals is too coarse to catch a single misbehaving Lambda before it's burned four figures — the granularity you detect at has to match the granularity a real incident happens at.

Use a rolling, seasonality-aware baseline instead of a static threshold. A flat "alert if daily spend > $X" rule either fires constantly during legitimate peaks (a marketing campaign, Black Friday) or stays silent during a real anomaly that happens during a naturally quiet period. A rolling baseline per service/account — comparing today's spend-by-hour against the same weekday's trailing average, with a tolerance band — catches genuine deviation from that resource's own normal pattern rather than from an arbitrary global number. This is the same instinct behind SLIs, SLOs, and Error Budgets: define "normal" from the system's own history, not a guessed constant.

Solve attribution with enforced tagging where you can, and a fallback heuristic where you can't. Tag policies and SCPs (or GCP Organization Policies) that block resource creation without a team/cost-center tag close the gap going forward, mirroring the centralized-least-privilege governance model in Pinterest: Resource Provisioner Pipeline. But untagged legacy resources still exist, so attribution needs a fallback: map by account ownership (if accounts are reasonably team-scoped) or by resource-creation IAM principal as a best-effort second signal, surfaced as "attributed" vs. "unattributed — needs owner" rather than silently guessing.

Tier alerts by severity and route to the owning team, not a central SRE inbox. A spike of $50/day on a $2,000/day baseline is noise; a spike of $50/day on a $10/day baseline is a real signal even though the absolute dollar amount is small — percentage deviation relative to baseline, not absolute dollars, is what should drive severity. Route confirmed anomalies straight to the owning team (via the attribution above) with the specific resource implicated, and reserve a central cost-governance channel for cross-account patterns or anything unattributed. For a small set of well-understood bad patterns (an idle dev/test instance left running over a weekend, a bucket with public-read egress), auto-remediate directly rather than paging a human — the same "known-bad-pattern gets automated, ambiguous case gets a human" split used in Chaos Engineering blast-radius containment.

Gaps to revisit

Detection latency is bounded by how quickly the cloud provider's billing data itself lands (AWS CUR can lag 24+ hours) — what's the fastest true "near-real-time" signal available (e.g., CloudWatch metrics as a proxy for spend before billing data catches up), and is it worth building a second, noisier fast path alongside the accurate slow path?
Multi-cloud estates multiply this problem — does a unified anomaly model even make sense across providers with different billing granularity and export formats, or does each cloud need its own detector feeding a shared alerting/attribution layer?
False-positive tuning is an ongoing tax — how do you measure and report "alert precision" over time so the system earns trust instead of getting muted by the teams it's supposed to warn?

Principal Engineer Lens

The tension underneath this system is the same one underneath most cost-governance work: perfect real-time detection and perfect attribution both cost real engineering effort, and neither is free relative to the dollars they save. Naming the ROI explicitly — "this catches spikes within 2 hours instead of 30 days, and the historical incidents we'd have caught total $X" — is what turns a cost-anomaly system from a nice-to-have into something finance will fund, and is exactly the kind of tradeoff-with-numbers argument a Principal Engineer needs to make in a budget review rather than an architecture review. It also generalizes past cost: any detection system built on a noisy, high-cardinality signal (cost, security events, latency) faces the same baseline-vs-threshold and precision-vs-recall tradeoffs, so getting fluent with this pattern here pays off well outside FinOps.

Reel Script

Setup: Your cloud bill jumps 40% overnight because of a misconfigured retry loop, and nobody notices until the invoice arrives three weeks later. How do you catch that in hours instead of weeks, across dozens of accounts with inconsistent tagging?

Concept walkthrough: Explain the pipeline — centralized billing export at fine granularity, a rolling seasonality-aware baseline instead of a flat threshold, tag-policy-enforced attribution with a heuristic fallback for legacy untagged resources, and severity-tiered routing straight to the owning team.

Real example tie-in: Walk through why a flat dollar threshold fails in both directions (false alarms during legitimate peaks, silence during real anomalies in quiet periods), and how percentage-deviation-from-baseline fixes both failure modes at once.

Tradeoffs & alternatives: Contrast a build-your-own anomaly pipeline against a managed offering (AWS Cost Anomaly Detection, CloudHealth, etc.) — the managed path is faster to stand up and good enough for many orgs, but a custom pipeline wins when you need tight integration with an internal attribution/ownership model that a generic tool can't see.

Principal Engineer takeaway: Cost governance systems succeed or fail on trust — an alert that's wrong twice gets muted forever. Optimizing for precision (few false alarms) over recall (catching every possible anomaly) early on is usually the right call, because a system nobody trusts is worse than no system at all.