Design a Multi-Tenant SaaS Metering & Cost Chargeback System
Scenario prompt
Design a system for a B2B SaaS platform that needs to meter per-tenant resource usage (API calls, storage, compute minutes) accurately enough to (a) generate customer invoices and (b) internally chargeback shared infrastructure costs to the product teams that drive them. Constraints:
- Usage events arrive from dozens of independent services, at high volume, with occasional duplicates and out-of-order delivery
- Billing must be auditable — every dollar on an invoice has to be traceable back to raw usage events, months after the fact, for support disputes and finance audits
- Tenants expect near-real-time usage dashboards (so they can see they're approaching a plan limit), even though the official billing run is a nightly batch job
- The system must survive an individual metering service crashing without losing usage (undercounting → revenue leakage) or double-counting it (overcounting → customer disputes)
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd reconcile the real-time dashboard path against the authoritative nightly billing numbers, and what you'd use as the source of truth.
Model solution
Ingestion — an append-only, idempotent event log, not a running counter. Every usage event (tenant_id, resource_type, quantity, timestamp, idempotency_key) gets written to a durable log (Kafka or equivalent) before anything aggregates it. The idempotency key lets duplicate delivery from a retrying producer get deduped deterministically instead of silently double-billing a tenant. Critically, the log — not any downstream aggregate — is the system of record: aggregates can be rebuilt from it, but the log itself is never rebuilt from an aggregate.
Two paths off the same log, deliberately inconsistent in different ways. A streaming path (windowed aggregation into a fast store like Redis or ClickHouse) feeds the tenant-facing dashboard — fast, eventually consistent, fine for a soft "you're near your limit" warning. A separate nightly batch job replays the log from scratch (or from the last checkpoint) to produce the authoritative invoice numbers. The dashboard is allowed to be approximately right in real time; the invoice is not allowed to be approximately right ever — it's recomputed from the immutable log, not trusted from the streaming aggregate.
Auditability drives a retention decision, not just a compliance checkbox. Since any invoice line item must be traceable back to raw events months later, raw usage events get archived to cold storage rather than deleted after aggregation. Corrections (a service double-reported usage, a bug undercounted for three days) are appended as new compensating events, never applied as in-place edits to historical records — the same event-sourcing discipline used in Event Sourcing and CQRS, applied here because "can we prove this invoice was correct" is a legal question, not just an engineering one.
One pipeline, two audiences. Tagging each usage event with the internal service/team that generated the underlying cost (not just the tenant who consumed it) lets the same event log drive customer billing and internal FinOps chargeback/showback — see FinOps: Cost Allocation, Tagging, Showback & Chargeback — without building and reconciling two separate metering systems.
Failure handling — checkpoints, not counts, are the resumable state. Aggregators track a log offset, not a running total, so a crash-and-restart replays from the last committed offset instead of guessing whether the last increment landed. This is what makes "survive a metering service crash without losing or double-counting" tractable at all.
Gaps to revisit
- Late-arriving events that straddle a billing period boundary — which invoice do they land on, and how is that policy communicated to the customer?
- Real-time usage caps: do you enforce a hard stop the moment a tenant crosses a limit (requires low-latency, strongly consistent checks) or only warn and true-up at billing time (simpler, but lets a tenant burst past their plan)?
- Discounts, credits, and free-tier overrides — where do they slot into a pipeline built around raw usage events without corrupting the audit trail?
- Multi-currency and tax-jurisdiction invoicing is a real complexity but probably belongs in a downstream billing system, not the metering pipeline itself — where's that boundary?
Principal Engineer Lens
The interesting move here is treating the billing pipeline like a ledger, not a metrics pipeline — the append-only, non-destructive event log is the same discipline as financial double-entry bookkeeping, and for the same reason: when someone disputes a number, "show me how you got there" has to be answerable from durable, replayable evidence, not from a mutable counter's current value. That distinction — a dashboard estimate is allowed to be wrong for a few seconds, an invoice is never allowed to be wrong — is exactly the kind of explicit consistency-tradeoff articulation that reads as Principal-level thinking in an architecture review. It's also a direct rehearsal for Fintech/Capital Markets-style problems: usage-based billing infrastructure and trade settlement systems both live or die on "can we reconstruct exactly how this number was produced," which is a much harder bar than "is this fast."
Reel Script
Setup: You're building the metering pipeline behind a SaaS product's invoices — every API call, GB stored, and compute-minute a tenant uses somehow has to turn into a number on a bill that a finance team can defend if a customer disputes it.
Concept walkthrough: Start from why a simple running counter per tenant breaks down — duplicate events double-bill, a crashed aggregator loses increments, and there's no way to explain how a number was reached six months later. The fix is treating raw usage events as an immutable, replayable log and computing both the real-time dashboard and the authoritative invoice as two different views over that same log, not as two independently-trusted counters.
Real example tie-in: Walk through the nightly batch job: it doesn't trust the streaming dashboard aggregate at all — it starts from the raw event log and recomputes billing numbers from scratch (or from a checkpoint), because the dashboard's job is speed and the invoice's job is correctness, and those are different guarantees.
Tradeoffs & alternatives: Contrast this against the tempting shortcut — just increment a counter per tenant per event — which is simpler and works fine until the first duplicate delivery or crash, at which point it silently produces wrong invoices with no way to audit or fix them. The event-log approach costs more storage and pipeline complexity in exchange for auditability that's non-negotiable once real money is involved.
Principal Engineer takeaway: Billing infrastructure is a good forcing function for learning to separate "what's fast" from "what's provably correct" — the same event-sourcing instinct that shows up anywhere a number has to be defended after the fact, from SaaS invoices to trade settlement.