Architecture/CaseStudies/cloudflare-security-insights-scaling-10x

Cloudflare: Scaling Security Insights to a 10x Increase in Global Scanning Capacity

Problem + constraints

Cloudflare's Security Insights system continuously scans every customer's assets and configurations to surface exploitable weaknesses. A scheduler publishes scan jobs onto a Kafka topic; those fan out to a fleet of specialized Go microservices ("checkers"), each responsible for scanning one asset or configuration type; each checker sends its findings to an internal API, which persists them in Postgres. Demand for more frequent scanning outgrew the pipeline: sustained throughput was stuck around 10 scans/second against a target of at least 100/second — a 10x gap — and the constraint was to close it without adding hardware.

Two things were quietly throttling the whole pipeline. Checkers consumed Kafka messages essentially one at a time, even though most message types didn't actually need strict in-order processing beyond their own scan's correctness. And on the API side, persistence queries were holding connections from the client-side Postgres connection pool open longer than necessary; under real request volume the pool exhausted, and every consumer downstream ended up blocked waiting for a free connection — backpressure propagating all the way back through the fan-out.

Solution

Two changes, both aimed at the actual bottlenecks rather than at adding capacity. First, Kafka consumption moved from strictly serial, one-message-at-a-time processing to batched consumption, with each message in a batch handled concurrently in its own goroutine — parallelizing work that didn't need to be serialized in the first place. Second, the API layer's Postgres query patterns and connection-pool usage were tightened so persistence calls held a connection for a shorter, bounded window instead of monopolizing the pool, directly relieving the backpressure that had been rate-limiting every checker upstream.

Together, these pushed sustained throughput from ~10 scans/second to over 120 scans/second at peak scheduling — past the 10x target — with zero additional servers. The ceiling had been architectural (serialized consumption + connection contention), not physical.

What to steal

Before reaching for more hardware, profile for "held resource, not held CPU" bottlenecks — a connection pool exhausted by queries that hold connections longer than necessary can throttle an entire fan-out pipeline as effectively as a CPU-bound stage would.
Serial message processing is a default, not a requirement. Check whether your real ordering/correctness constraint is per-key (needs order within a key) or global (doesn't) — then batch and parallelize within whatever scope is actually safe.
A 10x throughput win from software changes alone, with no new hardware, is a strong cost/performance story for a review: it's proof the ceiling was in the code, not the fleet.
Fan-out pipelines (scheduler → queue → many workers → shared datastore) concentrate bottlenecks at the shared datastore and its client library — that's the first place to look when throughput plateaus, before the workers themselves.

Principal Engineer Lens

This is a good example of a pillar-performance win that's really also a pillar-cost story — closing a 10x throughput gap without adding hardware is a directly quantifiable cost-avoidance argument, which is exactly the kind of framing that lands in a budget-constrained architecture review ("here's the throughput ceiling, here's why it was architectural, here's the fix that didn't require new spend"). The specific antipattern — a shared connection pool becoming the real bottleneck behind an apparently unrelated queue-consumer slowdown — is a pattern worth being able to recognize on sight; it shows up anywhere a fan-out pipeline terminates in a shared, stateful resource. It also has a genuine, non-forced analog to Mihir's network-tooling context: a scheduler polling or scanning a large device fleet (NetBox, Aegis) through a constrained queue-and-database backend is structurally the same shape, and the same two questions — "is this serialized when it doesn't need to be?" and "is something upstream holding a shared resource too long?" — apply directly.

Reel Script

Setup: Cloudflare's Security Insights system scans every customer's assets and configurations for security weaknesses, but throughput had plateaued around 10 scans/second when the target was at least 100/second — and the team couldn't just throw more servers at it.

Concept walkthrough: Walk through the pipeline — scheduler publishes to Kafka, checkers consume and scan, results go to an internal API, API persists to Postgres. Explain the two hidden bottlenecks: checkers processing Kafka messages one at a time when most didn't need strict ordering, and the API's Postgres queries holding connection-pool connections open long enough to exhaust the pool under load, which backed up pressure through the entire fan-out.

Real example tie-in: Walk through the fix — batch Kafka consumption with per-message goroutines for anything that's safe to parallelize, and tighter Postgres query/connection-pool discipline so persistence calls release connections quickly. Throughput went from ~10/sec to 120+/sec at peak — past the 10x goal — with no new hardware.

Tradeoffs & alternatives: Compare to the "just add more checkers/hardware" alternative — it would have masked the real bottleneck (the shared connection pool) rather than fixing it, and cost would have scaled with load indefinitely. Fixing the actual constraint scales for free from here; adding hardware would have needed to scale again at the next order of magnitude.

Principal Engineer takeaway: When a fan-out pipeline plateaus, look at what's shared and stateful at the far end of the fan-out before scaling the fan-out itself — that shared resource, and whatever's needlessly serialized upstream of it, is usually the real ceiling.

Sources: