Architecture/CaseStudies/slack-whitecastle-cloud-network-rearchitecture

Slack: Whitecastle — Rebuilding Slack's Cloud Network to Escape the VPC Peering Trap

Problem + constraints

Slack's cloud network grew the way most companies' do: one AWS account per team or service, each account allocating its own VPC CIDR range without central coordination. That worked fine until it didn't — once Slack had hundreds of AWS accounts, CIDR ranges started overlapping across accounts, and AWS VPC peering (the normal, boring way to connect two VPCs) simply doesn't work between VPCs with overlapping IP ranges. As a workaround, Slack reached for an AWS Transit Gateway to route around the peering problem, without fixing the underlying cause: IP space had never been centrally managed.

That stopgap became a hard dependency, and the cost showed up on January 4th, 2021: the first heavy-traffic Monday after the holidays, with client caches cold and every client pulling more data than usual on reconnect, one of Slack's Transit Gateways got overloaded and didn't scale fast enough to absorb the spike. It got worse because Slack's own dashboards and alerting services ran in a different VPC than their backend databases — meaning the tools meant to tell engineers what was happening were themselves stuck behind the same overloaded gateway. AWS ultimately had to manually intervene to scale TGW capacity.

The fix needed to solve the root cause (uncoordinated IP space across hundreds of accounts, in multiple AWS regions) without a risky big-bang cutover that could itself trigger the kind of outage it was meant to prevent.

Solution

Slack built Whitecastle, a ground-up redesign of its cloud network:

Shared VPCs to centralize IP address space allocation across all accounts, replacing the old model where every account picked its own CIDR range independently.
AWS Transit Gateway inter-region peering, done as the actual backbone of the network this time — not a patch bolted onto a broken peering model — connecting new VPCs across all of Slack's AWS regions.
A dedicated management VPC with connectivity to every other VPC, hosting shared management services (e.g., their Chef configuration management server).
Three environments — sandbox, dev, and prod — built out under the new model, with automated network testing continuously validating cross-region connectivity rather than trusting it silently.

The migration itself was the hard part. Instead of a big-bang cutover, Slack shared the new Transit Gateway back to the legacy child accounts and attached it to each region's old VPC — so as individual services moved onto the new Whitecastle VPCs, they could still reach whatever hadn't migrated yet, and vice versa. This let migration proceed incrementally, service by service, instead of requiring the whole company to cut over at once. Progress was tracked with a deliberately simple metric: a daily count of IP addresses and elastic network interfaces still live in each legacy VPC, across every AWS account — a plain proxy for "how much is actually left." Once production traffic was running on Whitecastle, the team locked down ad hoc changes to the new environment, treating it as load-bearing infrastructure rather than a project still in flux.

What to steal

The real defect wasn't "a Transit Gateway got overloaded" — it was that IP space was never centrally allocated. CIDR collisions across independently-provisioned accounts are what made ordinary VPC peering stop being an option at all; the TGW overload was a downstream symptom of a workaround for that root cause, not the root cause itself. Any resource allocated account-by-account or team-by-team without central coordination will eventually collide exactly where those units need to interoperate.
A stopgap fix reached for under pressure can quietly become a permanent, unexamined single point of failure. Nobody decided "the TGW is now critical path for the whole company" — it became that by default because it was the only way to route around a peering problem nobody had fixed properly. Stopgaps deserve a follow-up: did this ever get the real fix, or is three years of load bearing now resting on a patch?
Put your monitoring and alerting plane's connectivity in a different failure domain than what it watches. Slack's dashboards and alerting living behind the same overloaded TGW as the backend they were supposed to be watching meant the outage cost them their own visibility exactly when they needed it most.
When a migration can't be one cutover, build the bridge first. Sharing the new Transit Gateway back into the legacy VPCs — so migrated and unmigrated services could still talk to each other — is what turned a company-wide flag-day risk into a service-by-service risk that could be paused, resumed, or rolled back independently.
A boring, literal counter (IPs + ENIs remaining per legacy VPC, checked daily) beats a vibe check for tracking a multi-account migration. Without it, "are we done yet" has no better answer than someone's opinion.

Principal Engineer Lens

This is the network-layer instance of a pattern that recurs in every domain: a decision made without central coordination — each account picks its own CIDR block, the way each team might pick its own schema convention or its own retry policy — works fine right up until the system's own growth makes that decision collide with itself. The interesting architecture-review question here isn't "should Slack have centralized IP allocation from day one" — hindsight always says yes to more centralization — it's "what decisions are we making today, account-by-account or team-by-team, that only become a problem once we're 3x bigger, and which of those is cheapest to centralize now versus fixing live during an outage." Spotting that your monitoring and alerting infrastructure shares a failure domain with the system it watches is a governance smell a Principal-level reviewer should catch on sight, independent of the underlying technology — it's the identical critique as running your paging system's database on the cluster it's meant to page you about. And the migration mechanics — bridging old and new rather than cutting over — is the transferable skill: the redesign itself is the easy part to sketch on a whiteboard; making it safe to roll out incrementally across hundreds of live accounts is the part that actually determines whether the project ships without becoming its own incident.

Reel Script

Setup: On January 4th, 2021, one of Slack's AWS Transit Gateways got overloaded on the first heavy-traffic Monday after the holidays and didn't scale fast enough — and Slack's own dashboards and alerting were stuck behind that same overloaded gateway, so the team lost visibility into the outage while it was happening.

Concept walkthrough: Explain how the TGW became a single point of failure in the first place: Slack had grown to hundreds of AWS accounts, each picking its own CIDR range independently, and once ranges started overlapping across accounts, standard VPC peering stopped being possible — Transit Gateway was reached for as a workaround, without fixing the underlying IP-space problem. Then introduce Whitecastle: shared VPCs to centralize IP allocation, TGW inter-region peering built properly as the intended backbone this time, and a management VPC hosting shared services with connectivity to everything.

Real example / case study tie-in: Walk the migration mechanics — sharing the new TGW into the legacy per-account VPCs so migrated and unmigrated services could still talk to each other, moving services over incrementally instead of a big-bang cutover, and tracking progress with a daily count of IPs and ENIs remaining in the legacy VPCs. End state: production traffic running on Whitecastle, ad hoc changes locked down once it became load-bearing infrastructure.

Tradeoffs & alternatives: Contrast the incremental "bridge old and new" approach (slower, but every step is independently reversible and doesn't require a flag day) against a hypothetical rip-and-replace cutover across hundreds of accounts (faster on paper, but exactly the kind of change that produces the next January 4th). Also note the tradeoff of centralizing IP allocation at all: it constrains individual teams' autonomy over their own account's networking in exchange for making the whole system operable at scale.