Architecture/CaseStudies/github-30x-capacity-rearchitecture

GitHub: Rearchitecting for 30x Capacity Under Agentic-AI Load Growth

Problem + constraints

Starting in late 2025, GitHub's traffic profile changed faster than any prior growth curve the platform had absorbed: AI-agent-opened pull requests went from roughly 4M/month in September 2025 to 17M/month by March 2026 (a 325% six-month increase), AI-driven commits reached 275M/week (~14x year-over-year), and GitHub Actions compute usage jumped from 500M minutes/week in 2023 to 2.1B minutes in a single week in early 2026. GitHub had already planned a 10x capacity increase in October 2025 — and by February 2026 that plan was already insufficient; leadership concluded they needed to design for 30x current scale.

The growth alone wasn't the core failure — it exposed a structural weakness. CTO Vlad Fedorov's public diagnosis named three compounding causes: rapid load growth, architectural coupling that let localized issues cascade across critical services, and an inability to selectively shed load from misbehaving clients. The clearest illustration was a February 9 incident: a config change dropped a cache-refresh TTL from 12 hours to 2 hours to expedite a model rollout, colliding with per-user settings data that had organically grown from a few bytes to kilobytes over years (a design originally chosen "for simplicity" back when there were few models and governance controls). The resulting load spike overwhelmed the authentication/user-management database cluster — and the response team's only real lever was binary: deny all requests, or let the infrastructure collapse. There was no way to shed load by service, user tier, or request pattern. A separate March 5 incident saw automated failover for a Redis cluster backing Actions job orchestration fail due to a latent config issue, leaving the cluster with no writable primary.

Solution

GitHub's response is structural, multi-quarter rearchitecture work, not a single fix:

Isolating shared dependencies. GitHub committed to further isolating key dependencies so that a failure in shared infrastructure (the auth/settings database cluster, the Actions Redis cluster) can't take down unrelated critical services like Git operations or Actions — directly targeting the "architectural coupling" Fedorov named as root cause.
Building real load-shedding capability. In direct response to the Feb 9 postmortem's binary deny-or-collapse finding, GitHub is building the ability to selectively shed load by service, client, user tier, or request pattern, rather than only having an all-or-nothing kill switch.
Migrating monolith traffic to Azure for elastic capacity. GitHub moved a growing share of its Ruby-on-Rails monolith's traffic onto Azure — both for vertical scaling headroom within a region and horizontal scaling across regions — explicitly to reduce blast radius from the monolith's historically concentrated footprint. Azure's share of monolith traffic rose from roughly 8% (February 2026) to 40% (May 2026), with Git traffic at 30% and repo replication at 99% by that point; Central US monolith traffic peaked around 45% on Azure by June. The ramp was paused for about a month after a May 21 stability incident and resumed June 17 under tighter stability gates — capacity migration itself being treated as a change requiring its own safety rails.
Accelerating the Ruby-on-Rails-to-Go migration. Performance- and scale-sensitive code paths are being migrated out of GitHub's ~2-million-line Rails monolith into Go faster than originally planned — a modernization effort that predates the crisis but was compressed under the realization that agentic workflows were arriving faster than the migration could naturally keep pace with.
Public, recurring accountability. GitHub committed to monthly availability reports (published through at least June 2026) detailing individual incidents, root causes, and remediation status — a transparency mechanism that also functions as internal pressure to keep the rearchitecture work visibly moving.

The work is ongoing and imperfect in the interim: a June 4 Copilot code-review outage (an incompatible dependency auto-consumed without a compatibility gate, averaging 81.6% failure rate and roughly 36,800 failed requests) and a June 8 incident (abusive automated traffic spiking unauthenticated request failures to around 17%, peaking near 34%) both landed after the rearchitecture effort was already underway — illustrating that structural fixes and new failure modes are surfacing concurrently, not sequentially.

What to steal

A capacity plan is a snapshot, not a commitment — revisit it against the actual growth curve. GitHub's 10x plan wasn't wrong when it was made; it became wrong within months because the input (agentic traffic growth) accelerated faster than any prior planning cycle. Treat capacity targets as living numbers, not settled scope.
"Deny everything or collapse" is a design gap, not an incident response strategy. If your only lever under load is a global kill switch, you don't yet have load shedding — you have an outage switch. Selective shedding (by tier, client, or pattern) needs to be built before the incident that requires it.
A "simple for now" design choice is a future coupling risk if the underlying data or traffic pattern grows. The per-user settings data that grew from bytes to kilobytes, masked for years by a long cache TTL, is a textbook case of technical debt that's invisible until an unrelated change (a shortened TTL) removes the thing that was hiding it.
Treat an internal migration (cloud, region, or capacity ramp) as a change requiring its own safety gates. GitHub paused its own Azure migration after a stability incident rather than pushing through — the migration meant to reduce blast radius was itself capable of causing an incident, and they treated it accordingly.

Principal Engineer Lens

This case study is valuable less for the specific fixes and more for how cleanly it demonstrates the gap between "we have enough capacity" and "we have resilient capacity." GitHub wasn't short on raw compute — the failures were architectural coupling and a missing load-shedding capability, both of which no amount of added capacity would have fixed on their own; more headroom just delays the day the coupling problem bites. That distinction is exactly what separates a Principal-level capacity conversation from an infrastructure-scaling one: the question isn't just "can we handle N times current load," it's "when a component fails under that load, does the failure stay contained, and can we choose who gets denied service instead of everyone." For anyone in a Fintech or Capital Markets context, where regulators and counterparties expect graceful degradation over total outages, GitHub's Feb 9 postmortem — architecture forced a binary choice under load — is the cautionary tale to bring into a resilience review: any critical shared dependency that lacks a selective-shedding mechanism is one traffic spike away from becoming exactly that binary choice.

Reel Script

Setup: GitHub's traffic exploded in late 2025 and early 2026 as AI agents started opening millions of PRs and burning billions of Actions minutes a month — growth so fast that a 10x capacity plan from October 2025 was already inadequate by February 2026, and a wave of outages exposed why raw capacity wasn't the real problem.

Concept walkthrough: Walk through the Feb 9 incident as the clearest illustration: a shortened cache TTL exposed per-user settings data that had quietly grown from bytes to kilobytes over years, overwhelming the auth database cluster, and the team's only lever was a global deny-all — no selective load shedding existed. Then walk through GitHub's structural response: isolating shared dependencies so failures don't cascade, building real load-shedding by service/tier/client, migrating monolith traffic to Azure for elastic and blast-radius-reducing capacity, and accelerating a Rails-to-Go migration for the hottest code paths.

Real example / case study tie-in: Trace the throughline from the March "architectural coupling" diagnosis to the June availability reports — a Copilot code-review outage from an unvetted dependency, and an abusive-traffic incident — showing that the rearchitecture is a multi-month, still-in-progress effort, not a fix that landed all at once, and that new failure modes keep surfacing even as the structural work proceeds.

Tradeoffs & alternatives: Contrast with simply buying more capacity (the initial 10x plan) versus fixing the coupling and shedding-capacity gap (the actual multi-quarter effort) — more capacity buys time but doesn't stop a cascading failure from taking down unrelated services; only decoupling and selective shedding do that. Also note GitHub paused its own Azure migration after a stability incident, showing even the fix itself needed guardrails.