Works/ResumeProjects/p1-war-story-vlan-change-outage-elastiflow

P1 War Story — The Silent Change That Took Down a Mission-Critical App

[!note] What this note is A reconstructed, sanitized narrative of the ElastiFlow-diagnosed P1 you mentioned — written in SRE-on-the-bridge voice, not postmortem-document voice, so you can hear the actual thinking pattern an experienced P1 handler runs on autopilot. Timestamps, team names, and specifics below are illustrative placeholders — swap them for your real incident's details when you use this for interview prep or the Catalyst Q&A. The point isn't the exact minute-by-minute; it's the reasoning order: impact → scope → layer → cause → fix → why-it-hid.

1. The page (T+0:00)

Your phone doesn't ring politely. PagerDuty fires, Slack #incidents lights up two seconds later, and the first message in the war room is never technical — it's business impact:

"Mission-critical trading app is throwing errors for a subset of users. Sev-1 declared. Bridge: [link]. IC: [you]."

As the person on the bridge, your first internal question is not "what changed." It's:

Who is actually affected, and what does "affected" mean in dollars/risk right now? (all users or a subset? all regions or one? orders failing, or just slow?)
Is this getting worse, staying flat, or self-recovering?

You ask the app team on the bridge two questions out loud, because everyone needs to hear the answer, not just you:

"Is this 100% of traffic or a slice? And is it erroring, or just slow?"

Answer comes back: "A slice — looks regional. Not fully down, but error rate is climbing on order acknowledgments."

That single sentence already tells an experienced SRE something important: a full outage and a partial/regional outage point you in different directions. A clean full-down usually means something binary broke (device down, BGP session dropped, cert expired). A partial, regional, climbing-error-rate pattern smells like routing, load-balancing, or a policy change that only touches some paths — which is a very different investigation than "is the switch up."

2. Ruling out the obvious, fast (T+0:02 – T+0:05)

Before touching any deep tooling, you run the cheap checks first — not because they're likely to find it, but because ruling them out in under 60 seconds keeps the war room from splitting attention:

L1/L2 sanity: any open hardware alarms, interface flaps, or CRC error counters ticking up on the relevant switches/routers? (Nothing. Interfaces are clean.)
L3 reachability: can you ping/traceroute the affected app tier from a healthy region? (Yes — reachable. This immediately rules out "link is down" and "route is missing.")
Compute/app health: is the app team seeing CPU/memory/pod-restart signals on their side? (No — app team confirms their instances are healthy; they're just seeing upstream connection issues to a downstream dependency.)

This is the moment most people without an SRE instinct get stuck: "reachable" at L3 gets treated as "the network is fine," and the investigation pivots entirely to the app — usually the wrong move. Reachability is a binary, coarse signal. It tells you the path exists, not that the path is behaving correctly for this specific traffic pattern. That gap — reachable but not healthy — is where L4 flow behavior and L7 protocol-level truth live, and it's exactly where this incident actually was.

3. Going to flow data instead of guessing (T+0:05 – T+0:12)

This is the pivot point of the whole incident, and it's the one worth narrating carefully because it's the actual skill: when L1-L3 checks come back clean but the symptom is still there, you stop asking "is it up" and start asking "is the traffic shaped the way it should be." That question can't be answered by ping. It needs flow-level visibility.

You pull up ElastiFlow, scoped to the affected region's egress/ingress interfaces, last 30 minutes, and immediately look for three things:

Volume — did traffic volume on the affected path drop, spike, or stay flat?
New/missing flows — are flows that existed 20 minutes ago simply gone? Are there flows now taking a path they never used to?
Flow duration and reset behavior — are TCP sessions completing normally, or dying mid-stream?

What you find: a set of flows that used to traverse Path A now aren't appearing on Path A at all — and they're not appearing on the expected failover path (Path B) either. They've effectively vanished for a subset of source subnets. That's not a capacity problem (volume elsewhere is normal) and it's not a hard outage (other flows on the same devices are fine) — it's selective. Something is filtering or misrouting a specific slice of traffic, cleanly enough that it doesn't trip a link-down alarm anywhere.

Selective + clean + no alarms is the signature of a policy object, not a hardware failure: an ACL, a route-map, a QoS class-map, a VLAN membership, a security-group/firewall rule.

4. Correlating against the change log (T+0:12 – T+0:16)

With "selective traffic filtering on a specific subnet/path, no hardware signal" as the working hypothesis, the next move is almost reflexive for anyone who's been burned by this before: check what changed recently on the devices in that path, before doing anything else. Not because you assume malice or incompetence — because the overwhelming majority of "clean, selective, no-alarm" network symptoms are self-inflicted by a recent change, and it's the cheapest hypothesis to confirm or kill.

Cross-referencing the change-management log against the device list in the affected path: a firewall rule-set update went out ~18 minutes before the first error spike, described as "cleanup of unused legacy ACL entries" for an unrelated decommission project. It touched the same firewall pair sitting in this app's path.

That's the "oh" moment on the bridge. Nobody flagged it as risky because:

It passed the standard post-change health check (basic reachability/ping — which, as established above, doesn't catch this class of failure)
It wasn't scoped as "affecting this app" in the change ticket, because the person making the change didn't know this app's traffic depended on that specific rule ordering
It was a "cleanup," which mentally gets deprioritized as low-risk compared to a "new" change

This is the exact gap the blast-radius problem exists to close: the change owner had no way to see, at change time, that this app's flow depended on that rule. A topology/dependency view (NetBox-style) would have shown the app's declared dependencies; it would not have shown this specific undocumented reliance on rule ordering — that only shows up in observed traffic, which is why flow data (ElastiFlow) is what actually caught it, not the source of truth.

5. Confirming causality, not just correlation (T+0:16 – T+0:20)

Timing correlation alone isn't proof — a good IC pushes for one more level of confirmation before declaring root cause, because acting on the wrong "obvious" cause under pressure is how a 10-minute P1 becomes a 40-minute one:

Pull the exact rule diff from the change ticket.
Check whether the removed/reordered entries match the 5-tuple (source IP/subnet, destination IP/subnet, port, protocol) of the flows that vanished in ElastiFlow.
They match: the "unused" ACL entry being cleaned up was, in fact, an implicit-allow that a broader downstream deny rule depended on being above it. Remove it, and the broader deny now catches traffic it was never supposed to.

Now it's not "the timing lines up," it's "the exact traffic that disappeared is exactly the traffic this exact rule change would have blocked." That's the bar for calling root cause on a bridge — timing plus mechanism, not timing alone.

6. The fix, and the part people skip narrating (T+0:20 – T+0:24)

The fix itself is almost anticlimactic — revert the rule-set change, confirmed via the same change-management pipeline (not a manual out-of-band edit, even under P1 pressure, because an undocumented emergency fix becomes tomorrow's mystery outage). ElastiFlow shows the missing flows reappear within the next polling interval. App team confirms error rate dropping in real time. Bridge stands down at T+0:26.

Total user-visible impact: a handful of minutes. But notice how much of that time was spent not fixing anything — it was spent narrowing down which layer the problem lived at. That's the actual shape of most P1s: the fix, once you know what it is, takes 60 seconds. Finding out what it is is the whole incident.

7. The developer-to-SRE translation — what was actually happening in your head

If you're coming at this from a developer standpoint, here's the mental model translated into terms that map to how you already think about debugging application code:

SRE move	Developer-world equivalent
"Reachable ≠ healthy" — don't stop at ping	A service returning HTTP 200 doesn't mean the response body is correct. Status code is L3-equivalent; payload correctness is L7-equivalent.
Go to flow data when reachability is clean but symptom persists	Reach for request-level tracing/logs when a health check passes but users report bad behavior — you need the actual transaction, not the liveness probe.
"Selective, clean, no alarms" → suspect a policy object	A bug that only reproduces for one input shape, cleanly, with no exception thrown → suspect a conditional/config branch, not a crash.
Check the change log before deep-diving from scratch	`git blame` / recent deploy log before assuming a novel bug — most production incidents are a recent change, not a long-dormant landmine.
Confirm causality via exact match (5-tuple vs. rule diff), not just timing	Confirm a regression by bisecting to the exact commit and diffing it against the failure, not just "it started after Tuesday's deploy."
Revert through the normal pipeline, not an out-of-band hotfix	Don't `kubectl edit` a live resource under pressure — patch through CI/CD even during an incident, or you've created an undocumented drift bug for later.

8. Why this incident is the strongest evidence for the Catalyst talk's architecture

This is worth stating plainly, because it's not a coincidence — it's the argument in miniature:

NetBox/Nautobot alone (structural blast radius) would not have caught this. The dependency wasn't in the documented topology — it was in undocumented rule ordering that only observed traffic could reveal.
A pure L3 reachability check would not have caught this. That's exactly why it went unnoticed by the automated post-change health check.
Flow-level telemetry (ElastiFlow) is what actually surfaced it — not because it's smarter, but because it's the only layer that shows what traffic is actually doing, as opposed to what the topology says it should do or whether the path is technically up.

That's the live, real-world instance of the fusion architecture from the earlier discussion: structural truth tells you what could be affected; flow truth tells you what actually is. This incident is proof the gap between those two is not theoretical — it cost real minutes on a mission-critical app, and the only reason it resolved in single-digit minutes instead of much longer is that someone thought to pivot to flow data instead of trusting the clean L3 reachability check.

9. Using this for interview / Q&A prep

When asked "tell me about a P1 you handled" or fielding a Catalyst Q&A skeptic, the shape to reuse:

Impact first — what broke, for whom, how bad (never lead with the technical cause).
The wrong turn everyone almost takes — "reachable, so we almost blamed the app" — this is the line that shows judgment, not just tooling knowledge.
The pivot — why flow-level data, specifically, was the right next tool, and what question it answered that L1-L3 checks couldn't.
Causality, not correlation — the 5-tuple match, not just "the timing lined up." This is the detail that separates a real incident handler from someone reciting a timeline.
The systemic fix, not just the tactical one — what changed afterward (e.g., change-review now requires a flow-impact check for firewall/ACL changes, not just a reachability check) — ties back to why an evidence-grounded triage agent is worth building at all.