Stripe Uses Graph Search and State Machines to Automate Database Remediation

Details

Stripe's engineering team published a detailed post-mortem of how they automated MongoDB database incident recovery. Their original plugin-based system used hard-coded, layout-specific logic that couldn't scale: in one six-month window it paged operators 124 times for misconfigured shards and 32 times for single-node-down scenarios, with critical operations like index builds blocked for an average of one hour per incident. The new system models the entire global MongoDB infrastructure as a directed graph — nodes represent infrastructure components, edges capture relationships, and node attributes describe current health state. Recovery plans are no longer hard-coded; instead, a graph traversal algorithm finds valid paths from the current broken state to a healthy goal state. Stripe initially used BFS to find recovery paths, then upgraded to Dijkstra's algorithm to prefer lower-cost plans and enable partial remediation when no complete path exists. Remediation logic is expressed as composable rules with explicit state transitions rather than fixed workflows, so the same planner handles different MongoDB shard layouts automatically. Results: database-related pager alerts dropped about 30%, translating to 200 fewer pages per year and roughly 12 fewer days of unhealthy shard states annually. The team plans to extend the framework to automate topology changes, blue-green deployments, and planned maintenance.