Developer/Observability/NetworkFlowMonitoring-NetFlow-sFlow-VPCFlowLogs/Fundamentals/netflow-sflow-vpc-flow-logs

NetFlow, sFlow, and VPC Flow Logs

Concept

Application logs and metrics record what your code thinks happened — a request was received, a query took 40ms. Network flow monitoring records what actually crossed the wire at the network layer, independent of whether the application ever logged anything about it, which makes it the only reliable source of truth for questions like "did this traffic even leave the instance" or "was this connection rejected by a security group before it ever reached the app." Three distinct mechanisms answer this at different layers:

NetFlow (Cisco-originated, now standardized as IPFIX) — the router/switch tracks and exports metadata for every flow (a unique 5-tuple of source/dest IP, source/dest port, and protocol) that crosses an interface: byte counts, packet counts, start/end time. It's exhaustive — every flow is accounted for — but that exhaustiveness costs the device CPU/memory to track session state for every active flow.
sFlow (InMon-originated) — instead of tracking every flow, the switch samples a subset of packets (e.g. 1-in-1000) and exports the full packet header for each sampled packet, plus periodic interface counters. Sampling means dramatically lower overhead on the device — well suited to very high-throughput switches — at the cost of statistical rather than exact traffic counts, and a real chance of missing small, infrequent flows entirely.
VPC Flow Logs (AWS's cloud-native equivalent; GCP and Azure have their own analogs) — captures IP traffic metadata for network interfaces inside a VPC: source/dest IP and port, protocol, byte/packet counts, and critically an ACCEPT/REJECT verdict per record, shipped to CloudWatch Logs or S3. It answers the cloud-specific version of the same question NetFlow/sFlow answer on physical routers, but as a managed service you enable per-VPC/subnet/ENI rather than configuring on hardware you own.

The three exist because the underlying problem — "prove what traffic actually happened at the network layer" — shows up in different infrastructure contexts: NetFlow/sFlow on physical or virtualized on-prem/hybrid network gear, VPC Flow Logs in a public cloud where you don't control (or need to configure) the underlying router hardware at all.

Tradeoffs

Mechanism	Coverage	Overhead on the device	Best for
NetFlow / IPFIX	Every flow tracked exactly	Higher — device maintains per-flow session state	Capacity planning, exact bandwidth accounting, on-prem/hybrid infra you control
sFlow	Statistical sample of packets	Very low — no session state, just sampled headers + counters	Very high-throughput switches where exhaustive tracking would be too costly; approximate traffic analysis
VPC Flow Logs	Every flow at the ENI/subnet/VPC level, cloud-managed	Negligible to you — AWS runs the capture; cost is in log storage/ingestion, not device CPU	Cloud-native security/connectivity debugging (why was this REJECTed) and audit, without owning any network hardware

The accuracy-vs-overhead tradeoff between NetFlow and sFlow is the same fundamental tension full accounting vs. sampling shows up as everywhere in monitoring (compare to histogram bucket cost, or full request logging vs. sampled tracing) — sFlow trades exactness for the ability to run at line rate on hardware that would otherwise be overwhelmed tracking every session. VPC Flow Logs sidesteps the tradeoff for cloud users specifically because AWS absorbs the capture cost as part of the managed network fabric, but it introduces its own cost dimension (log storage volume and CloudWatch/S3 ingestion charges at high traffic levels).

When to use / when not to

In a public cloud environment, enable VPC Flow Logs by default at the VPC level for security and audit purposes — it's the fastest way to confirm whether a connectivity problem is a security group/NACL rejection versus something failing above the network layer, and the ACCEPT/REJECT field answers that in one query.
When debugging a connectivity issue in AWS, query flow log REJECT entries first — if there are none, the network layer let the traffic through and the problem is upstream in the application; if there are REJECTs, a security group or NACL is the blocker and you don't need to look anywhere else.
Use NetFlow when you control the network hardware (on-prem, hybrid, colo) and need exact bandwidth accounting or capacity planning data — identifying which applications or hosts are consuming the most bandwidth.
Use sFlow instead of NetFlow specifically when the switches involved operate at a scale where full per-flow session tracking would itself become a performance bottleneck on the device — very high port-count, high-throughput data center switches are the classic case.
Don't expect to configure NetFlow or sFlow directly on managed cloud infrastructure — they're protocols for hardware/software you administer directly; in AWS/GCP/Azure, the cloud provider's own flow-log service is the equivalent surface.

Common pitfall

Assuming flow logs (of any kind) capture packet payloads or give application-level detail — they capture metadata only (who talked to whom, on what port, how much, accept or reject). A REJECT-heavy investigation into a security group rule works well with flow logs; diagnosing why an HTTP request inside an accepted connection returned a 500 requires application logs and tracing, not flow data. Treating flow logs as a substitute for application observability rather than a complement to it leads to spending time in the wrong tool for the question being asked.

Engineering Lens

The operational-excellence question flow monitoring actually answers in an incident is narrower and more useful than it first appears: not "what happened," but "did this traffic reach the network boundary it was supposed to reach, and was it allowed through." That's a question application logs structurally cannot answer, because an application only logs what it received — if a security group silently dropped a request before the app ever saw it, there is no application log entry to find, only a flow-log REJECT. The transferable lesson for any infrastructure review is to ask, for a given VPC/network segment, whether flow logging is actually enabled before an incident requires it — discovering flow logs were never turned on is a bad moment to discover it, since they can't be retroactively generated for traffic that already passed.