Architecture/CaseStudies/stripe-mongodb-fleet-auto-remediation

Stripe: Graph Search and State Machines to Auto-Remediate a Global Database Fleet

Problem + constraints

Stripe runs a global fleet of 2,000+ sharded MongoDB clusters underpinning payment processing. Shards degrade in ordinary, recurring ways — a replica falls behind, a node needs replacement, a config server drifts out of sync — and each degraded state used to page an on-call engineer. That added up to hundreds of pages a year, and manual recovery from a single degraded shard could take up to 3 hours of hands-on work: diagnose which specific thing was wrong, recall (or look up) the right sequence of operations to fix it, execute them in order, and confirm the shard came back healthy. The playbook wasn't complicated in any single case, but the combinatorics were — many different broken states, many different valid repair sequences, and a fleet large enough that "wrong shard, wrong time" incidents were a matter of when, not if. The team also needed the fix to generalize: new shard layouts and topologies get introduced over time, and a solution that only codified today's known-bad states would need constant manual upkeep to stay current.

Solution

Stripe modeled the MongoDB fleet's operational states as a traversable graph rather than as a fixed list of runbooks. Nodes in the graph represent possible infrastructure states (a shard's topology, replica health, and configuration); edges represent valid, individually-safe operations that move a shard from one state toward another (add a replica, promote a member, drain a node, rebalance a config server, etc.). Recovery stops being "look up the runbook for this specific failure" and becomes a pathfinding problem: given a shard's current (unhealthy) state and the graph of legal transitions, compute the shortest safe path back to a healthy state, the same way a routing algorithm finds a path across a road network. A state machine then executes that computed path step by step — each transition is its own bounded, idempotent operation, checked and retried independently, rather than one long fragile manual sequence.

The generality is the payoff: because recovery is derived from the graph structure rather than hardcoded per failure mode, a new shard layout or topology just adds nodes and edges to the graph — the pathfinding and execution logic doesn't change. The system now self-heals across all 2,000+ shards with zero manual effort for new layouts, cutting pager volume by roughly 30% (~200 pages/year) and eliminating about 12 days of cumulative unhealthy-shard-state per year fleet-wide.

What to steal

When an operational domain has many failure states and many valid repair sequences, look for whether it's secretly a graph-search problem before writing more runbooks. Runbooks enumerate; graphs generalize — a graph handles states nobody explicitly wrote a runbook for, as long as the legal transitions are modeled correctly.
Decompose "recovery" into small, independently idempotent, retryable operations (graph edges) rather than one long imperative script. That's what makes a computed multi-step plan safe to execute unattended — any single step can fail and retry without corrupting the overall state.
Design the automation to absorb new topology without new code. A graph that only needs new nodes/edges to support a new shard layout scales with the business; a runbook library that needs a new document per layout doesn't.
Measure toil reduction in the units that matter to the team doing the toil — pages avoided and hours of unhealthy state eliminated — not just "automation coverage %." That's what makes the ROI legible to a Principal-level audience deciding whether to invest further.

Principal Engineer Lens

This is a clean example of turning an operational-excellence problem (on-call toil, inconsistent manual recovery) into a resilience investment with a measurable payback — the kind of cross-pillar framing that reads well in an architecture review: "here's the toil cost today, here's the automation investment, here's the pager-load and downtime-days it buys back." It also generalizes well beyond databases: any infrastructure domain with enumerable states and known-legal transitions (network device configuration drift, certificate rotation, container placement) is a candidate for the same graph-search-plus-state-machine shape instead of an ever-growing pile of if/else remediation scripts — a genuinely close analogy to network topology remediation and config-drift correction, the same category of problem Mihir's NetBox/Aegis tooling exists to manage. Being able to recognize "this is a graph problem wearing a runbook's clothing" is a reusable Principal-level pattern-matching skill, not a MongoDB-specific one.

Reel Script

Setup: Stripe's on-call engineers were getting paged hundreds of times a year for degraded MongoDB shards, and fixing each one manually could eat up to 3 hours — not because any single fix was hard, but because there were so many different ways a shard could be broken and so many valid repair sequences to remember.

Concept walkthrough: Explain the reframe — instead of writing a runbook per failure mode, model every possible shard state as a node in a graph and every safe operation (add replica, promote member, rebalance config server) as an edge. Recovering a broken shard becomes a pathfinding problem: find the shortest safe path from "broken" to "healthy" through that graph, the same way a maps app routes you across a road network. A state machine then walks that computed path step by step, executing each small idempotent operation and confirming it before moving to the next.

Real example tie-in: Walk through a shard stuck with a lagging replica and a drifted config server — the old way meant an engineer diagnosing both problems and manually sequencing the fix; the new way computes the path automatically and self-heals it, one of 2,000+ shards doing this fleet-wide with zero manual involvement.

Tradeoffs & alternatives: Compare to the runbook-library alternative — cheaper to start (write a doc per known failure), but it doesn't generalize: every new topology or shard layout needs a new runbook, and coverage always lags behind what the fleet actually does. The graph approach costs more up front (you have to correctly model all legal states and transitions) but scales with the business — a new layout just adds graph structure, not new remediation code.

Principal Engineer takeaway: Look for the graph hiding inside any domain with lots of failure states and lots of valid repair paths — config drift, cert rotation, container placement, network topology remediation all fit the same shape. Recognizing that pattern and building the graph once is a much stronger long-term bet than writing the hundredth runbook.

Sources: