Stripe: Graph Search and State Machines to Auto-Remediate a Global Database Fleet
Problem + constraints
Stripe runs a global fleet of 2,000+ sharded MongoDB clusters underpinning payment processing. Shards degrade in ordinary, recurring ways — a replica falls behind, a node needs replacement, a config server drifts out of sync — and each degraded state used to page an on-call engineer. That added up to hundreds of pages a year, and manual recovery from a single degraded shard could take up to 3 hours of hands-on work: diagnose which specific thing was wrong, recall (or look up) the right sequence of operations to fix it, execute them in order, and confirm the shard came back healthy. The playbook wasn't complicated in any single case, but the combinatorics were — many different broken states, many different valid repair sequences, and a fleet large enough that "wrong shard, wrong time" incidents were a matter of when, not if. The team also needed the fix to generalize: new shard layouts and topologies get introduced over time, and a solution that only codified today's known-bad states would need constant manual upkeep to stay current.
Solution
Stripe modeled the MongoDB fleet's operational states as a traversable graph rather than as a fixed list of runbooks. Nodes in the graph represent possible infrastructure states (a shard's topology, replica health, and configuration); edges represent valid, individually-safe operations that move a shard from one state toward another (add a replica, promote a member, drain a node, rebalance a config server, etc.). Recovery stops being "look up the runbook for this specific failure" and becomes a pathfinding problem: given a shard's current (unhealthy) state and the graph of legal transitions, compute the shortest safe path back to a healthy state, the same way a routing algorithm finds a path across a road network. A state machine then executes that computed path step by step — each transition is its own bounded, idempotent operation, checked and retried independently, rather than one long fragile manual sequence.
The generality is the payoff: because recovery is derived from the graph structure rather than hardcoded per failure mode, a new shard layout or topology just adds nodes and edges to the graph — the pathfinding and execution logic doesn't change. The system now self-heals across all 2,000+ shards with zero manual effort for new layouts, cutting pager volume by roughly 30% (~200 pages/year) and eliminating about 12 days of cumulative unhealthy-shard-state per year fleet-wide.
What to steal
- When an operational domain has many failure states and many valid repair sequences, look for whether it's secretly a graph-search problem before writing more runbooks. Runbooks enumerate; graphs generalize — a graph handles states nobody explicitly wrote a runbook for, as long as the legal transitions are modeled correctly.
- Decompose "recovery" into small, independently idempotent, retryable operations (graph edges) rather than one long imperative script. That's what makes a computed multi-step plan safe to execute unattended — any single step can fail and retry without corrupting the overall state.
- Design the automation to absorb new topology without new code. A graph that only needs new nodes/edges to support a new shard layout scales with the business; a runbook library that needs a new document per layout doesn't.
- Measure toil reduction in the units that matter to the team doing the toil — pages avoided and hours of unhealthy state eliminated — not just "automation coverage %." That's what makes the ROI legible to a Principal-level audience deciding whether to invest further.
Engineering Lens
This is a clean example of turning an operational-excellence problem (on-call toil, inconsistent manual recovery) into a resilience investment with a measurable payback — the kind of cross-pillar framing that reads well in an architecture review: "here's the toil cost today, here's the automation investment, here's the pager-load and downtime-days it buys back." It also generalizes well beyond databases: any infrastructure domain with enumerable states and known-legal transitions (network device configuration drift, certificate rotation, container placement) is a candidate for the same graph-search-plus-state-machine shape instead of an ever-growing pile of if/else remediation scripts — a genuinely close analogy to network topology remediation and config-drift correction, the same category of problem Mihir's NetBox/Aegis tooling exists to manage. Being able to recognize "this is a graph problem wearing a runbook's clothing" is a reusable Principal-level pattern-matching skill, not a MongoDB-specific one.