Architecture/CaseStudies/discord-scylla-control-plane-automation

Discord: Automating ScyllaDB Cluster Operations with the Scylla Control Plane

Problem + constraints

Discord's Persistence Infrastructure team — seven engineers — runs the ScyllaDB layer underneath messages, channels, servers, and most of Discord's user data: over 20 clusters, close to 500 nodes combined. Cluster-wide operations like rolling OS/ScyllaDB upgrades, cluster expansion, node recovery, and standing up a shadow cluster (a full replica that mirrors production traffic, used to safety-test a new ScyllaDB release before it touches real traffic) were historically driven by a pile of ad hoc Python and bash scripts. The tooling worked, but only in the hands of the specific engineers who remembered every landmine — undocumented ordering dependencies, manual precondition checks, no built-in retry or resume if a multi-hour operation failed partway through. Standing up a full shadow cluster took roughly a day and a half of careful, error-prone manual work. For a team of seven responsible for the data layer under the entire platform, the bus factor on safely running these operations was uncomfortably thin.

Solution

Discord rebuilt cluster operations as the Scylla Control Plane (SCP), a task automation and orchestration framework purpose-built for safe, cluster-wide ScyllaDB workflows. Operations — rolling upgrades, cluster expansion, shadow-cluster provisioning, node recovery — are encoded as YAML-defined workflows built from small, composable task primitives, each with explicit retry counts, parallelism controls, zone-aware batching, per-step precondition checks, and abort-on-failure semantics. Idempotency is a hard requirement for every task, so any step that fails mid-workflow can simply retry, or the whole workflow can resume from where it left off, without risking a half-applied, inconsistent cluster state.

That resumability is what turns a fragile, must-babysit, multi-hour manual sequence into something the team can kick off and walk away from. Standing up a full shadow cluster now takes under two hours, down from a day and a half.

What to steal

Name the actual failure mode of ad hoc ops scripts precisely: it's not that they're wrong, it's that they encode tribal knowledge in individual engineers' heads instead of in the system — bus factor, not correctness, becomes the real constraint on operational safety.
Idempotency as a hard requirement for every task — not a nice-to-have — is what makes "retry" and "resume" safe defaults for long, multi-step, cluster-wide operations. Non-idempotent steps force you back into "diagnose exactly where it broke by hand," which is the toil the automation was supposed to remove.
YAML-defined workflows built from named, composable task primitives give you an audit trail and a shared vocabulary — "this workflow ran these tasks in this order" — that ad hoc scripts never provide, which matters directly in an incident review.
A roughly 9x reduction in the cost of a safety-testing step (shadow-cluster provisioning) doesn't just save time — it changes team behavior. Expensive safety nets get skipped under deadline pressure; cheap ones get used every time.

Principal Engineer Lens

The transferable idea here is small-team leverage over a large, stateful fleet through workflow automation and strict idempotency — not a ScyllaDB-specific trick. Any domain with expensive, error-prone, multi-step operations against a large fleet (rolling firmware or OS upgrades across a device fleet, coordinated config changes across many nodes, staged recovery procedures) is a candidate for the same shape: composable idempotent primitives, declarative workflow definitions, and resumability as a first-class property rather than an afterthought. That has a genuine, specific tie to Mihir's Network tooling — a multi-step device provisioning or upgrade workflow across a fleet managed through NetBox/Aegis is structurally the same problem SCP solves, and "is every step idempotent enough to retry blind?" is the right first question to ask of that kind of automation, in either domain.

Reel Script

Setup: Discord's entire messages/channels/user-data layer runs on ScyllaDB — over 20 clusters, nearly 500 nodes — operated by a team of just seven engineers, using a pile of ad hoc Python and bash scripts where cluster-wide operations depended on a handful of people remembering every landmine by heart.

Concept walkthrough: Explain the rebuild — the Scylla Control Plane encodes operations like rolling upgrades and shadow-cluster provisioning as YAML-defined workflows made of small, composable task primitives: retry counts, parallelism controls, zone-aware batching, precondition checks, abort-on-failure. The critical design constraint is that every task must be idempotent, so any step — or the whole workflow — can be safely retried or resumed after a failure instead of requiring a human to diagnose exactly where things broke.

Real example tie-in: Walk through shadow-cluster provisioning specifically — standing up a full replica of production to safety-test a new ScyllaDB release used to take a day and a half of hands-on work; with SCP it's under two hours, because the workflow can just be launched and left to run.

Tradeoffs & alternatives: Compare to staying with ad hoc scripts — cheaper to write initially, but the cost shows up later as bus factor and incident risk: only a few people can safely run the riskiest operations, and a failure partway through means manual diagnosis under pressure. SCP costs more up front (every task has to genuinely be made idempotent, which is real design work) but converts operational risk into a repeatable, auditable, low-toil process.

Principal Engineer takeaway: When a small team owns a large stateful fleet, the leverage isn't more headcount — it's turning tribal-knowledge runbooks into idempotent, composable, resumable workflows. The same shape applies well beyond databases, to any fleet-wide operation that's currently "safe only because so-and-so knows how to do it."

Sources: