Netflix: Building a Real-Time Service Topology Map
Problem + constraints
Netflix runs thousands of microservices, and for years the answer to "what talks to what, and how healthy is that path right now" lived in silos: distributed tracing told you about individual requests, IPC client metrics told you about call volumes between services that knew about each other, and network-level visibility was largely absent. None of those sources alone could answer an incident-response question like "what depends on this database, transitively, right now" — tracing samples individual requests rather than mapping the whole graph, and service-reported metrics miss anything happening below the application layer (load balancers, NAT gateways, API gateways, proxies) that silently sits between two services and obscures the real caller. Constraints: the map has to be near-real-time (stale topology is actively misleading during an incident), it has to reconstruct genuine application-to-application edges even when hidden behind network intermediaries, and it has to hold up at a throughput of millions of flow records per second across multiple AWS regions without falling behind.
Solution
Netflix built Service Topology as a three-stage distributed aggregation pipeline processing eBPF-captured network flow logs, consumed off Kafka, using Apache Pekko Streams (the open-source fork of Akka) for fault-tolerant, backpressured stream processing. Stage 1 does initial aggregation straight off Kafka. Stage 2 is the clever part: resolution logic that identifies known network intermediaries — load balancers, NAT gateways, API gateways, proxies — and stitches their separately-observed incoming and outgoing flows back together to reconstruct the direct application-to-application edge that the intermediary was obscuring. Stage 3 does final aggregation and merges in health-status signals before writing the result to a graph database (built on Netflix's own distributed key-value storage) optimized for fast multi-hop traversal. The pipeline auto-partitions work across Auto Scaling Groups to absorb volume, and the whole system is designed as physically separate graph layers — eBPF flows, IPC metrics, tracing — that can be queried independently or merged into one composite view, rather than forcing every consumer through a single fused representation. End result: sub-second query responses against a graph that stays close to real-time, at millions of flow records per second.
What to steal
- Don't rely on one signal source (tracing, or self-reported service metrics) to build a topology map — each has structural blind spots (sampling in tracing; anything an intermediary hides in service-reported metrics), and combining independently-collected signals catches what any single source misses.
- The resolution step — explicitly modeling known network intermediaries so their split incoming/outgoing flows can be recombined into the real edge — is the difference between a topology map that shows "everything talks to the load balancer" (useless) and one that shows the actual dependency graph (useful). Naive flow aggregation without this step produces a graph centered on infrastructure nodes instead of application relationships.
- Keep the underlying data layers physically separate and independently queryable rather than fusing everything into one schema upfront — it lets different consumers pick the view (or combination of views) they actually need, and lets each layer evolve or fail independently.
- A streaming aggregation pipeline that auto-partitions across an Auto Scaling Group with natural backpressure is a reusable shape for any "reduce a firehose into a queryable summary" problem, not just topology mapping.
Engineering Lens
Real-time topology is what turns "we think X depends on Y" into "we can prove X depends on Y, transitively, as of 10 seconds ago" during an incident review — the difference between tribal knowledge and a queryable source of truth is exactly the kind of infrastructure investment a Principal Engineer has to justify against its build cost. The specific technique of reconstructing hidden edges behind network intermediaries generalizes well beyond Netflix's AWS/eBPF stack: any architecture review that asks "what's the actual blast radius if this service goes down" is implicitly asking for this graph, and being able to describe how you'd build or query one (rather than hand-waving "we'd trace it") signals the kind of systems thinking expected at Principal level, in fintech/capital-markets contexts just as much as consumer platforms.