Architecture/CaseStudies/netflix-real-time-service-topology

Netflix: Building a Real-Time Service Topology Map

Problem + constraints

Netflix runs thousands of microservices, and for years the answer to "what talks to what, and how healthy is that path right now" lived in silos: distributed tracing told you about individual requests, IPC client metrics told you about call volumes between services that knew about each other, and network-level visibility was largely absent. None of those sources alone could answer an incident-response question like "what depends on this database, transitively, right now" — tracing samples individual requests rather than mapping the whole graph, and service-reported metrics miss anything happening below the application layer (load balancers, NAT gateways, API gateways, proxies) that silently sits between two services and obscures the real caller. Constraints: the map has to be near-real-time (stale topology is actively misleading during an incident), it has to reconstruct genuine application-to-application edges even when hidden behind network intermediaries, and it has to hold up at a throughput of millions of flow records per second across multiple AWS regions without falling behind.

Solution

Netflix built Service Topology as a three-stage distributed aggregation pipeline processing eBPF-captured network flow logs, consumed off Kafka, using Apache Pekko Streams (the open-source fork of Akka) for fault-tolerant, backpressured stream processing. Stage 1 does initial aggregation straight off Kafka. Stage 2 is the clever part: resolution logic that identifies known network intermediaries — load balancers, NAT gateways, API gateways, proxies — and stitches their separately-observed incoming and outgoing flows back together to reconstruct the direct application-to-application edge that the intermediary was obscuring. Stage 3 does final aggregation and merges in health-status signals before writing the result to a graph database (built on Netflix's own distributed key-value storage) optimized for fast multi-hop traversal. The pipeline auto-partitions work across Auto Scaling Groups to absorb volume, and the whole system is designed as physically separate graph layers — eBPF flows, IPC metrics, tracing — that can be queried independently or merged into one composite view, rather than forcing every consumer through a single fused representation. End result: sub-second query responses against a graph that stays close to real-time, at millions of flow records per second.

What to steal

Don't rely on one signal source (tracing, or self-reported service metrics) to build a topology map — each has structural blind spots (sampling in tracing; anything an intermediary hides in service-reported metrics), and combining independently-collected signals catches what any single source misses.
The resolution step — explicitly modeling known network intermediaries so their split incoming/outgoing flows can be recombined into the real edge — is the difference between a topology map that shows "everything talks to the load balancer" (useless) and one that shows the actual dependency graph (useful). Naive flow aggregation without this step produces a graph centered on infrastructure nodes instead of application relationships.
Keep the underlying data layers physically separate and independently queryable rather than fusing everything into one schema upfront — it lets different consumers pick the view (or combination of views) they actually need, and lets each layer evolve or fail independently.
A streaming aggregation pipeline that auto-partitions across an Auto Scaling Group with natural backpressure is a reusable shape for any "reduce a firehose into a queryable summary" problem, not just topology mapping.

Principal Engineer Lens

Real-time topology is what turns "we think X depends on Y" into "we can prove X depends on Y, transitively, as of 10 seconds ago" during an incident review — the difference between tribal knowledge and a queryable source of truth is exactly the kind of infrastructure investment a Principal Engineer has to justify against its build cost. The specific technique of reconstructing hidden edges behind network intermediaries generalizes well beyond Netflix's AWS/eBPF stack: any architecture review that asks "what's the actual blast radius if this service goes down" is implicitly asking for this graph, and being able to describe how you'd build or query one (rather than hand-waving "we'd trace it") signals the kind of systems thinking expected at Principal level, in fintech/capital-markets contexts just as much as consumer platforms.

Reel Script

Setup: At Netflix's scale, "what depends on this service" isn't a question you can answer by asking engineers — you need a live map, and existing signals (tracing, service-reported metrics) each have blind spots that make them individually insufficient.

Concept walkthrough: Walk through the three-stage pipeline: Stage 1 aggregates eBPF flow logs off Kafka, Stage 2 resolves network intermediaries (load balancers, gateways, proxies) by stitching their split incoming/outgoing flows back into direct application edges, Stage 3 merges in health status and persists to a graph database built for fast multi-hop traversal. Emphasize why Stage 2 is the hard part — without it, the graph just shows everything pointing at infrastructure nodes instead of real dependencies.

Real example tie-in: Picture an incident where a database is degraded — querying the live topology graph answers "what's transitively affected right now" in a sub-second query, instead of paging through dashboards or asking around, because the graph is reconstructed from real traffic, not a stale architecture diagram.

Tradeoffs & alternatives: Contrast with relying purely on distributed tracing (samples individual requests, doesn't give you the full graph) or purely on service-reported IPC metrics (misses anything an intermediary hides). Building and maintaining a dedicated streaming pipeline plus a graph database is real infrastructure cost — justified at Netflix's scale of thousands of services, harder to justify for a ten-service startup where a static diagram might genuinely be good enough.

Principal Engineer takeaway: When someone asks "what breaks if this goes down," the mature answer is a system that can prove it from live traffic, not a diagram someone drew eighteen months ago. Knowing how that system would be built — multi-source aggregation, intermediary resolution, graph storage — is what lets you scope that investment credibly in an architecture review.