Cloudflare: Scaling Security Insights to a 10x Increase in Global Scanning Capacity
Problem + constraints
Cloudflare's Security Insights system continuously scans every customer's assets and configurations to surface exploitable weaknesses. A scheduler publishes scan jobs onto a Kafka topic; those fan out to a fleet of specialized Go microservices ("checkers"), each responsible for scanning one asset or configuration type; each checker sends its findings to an internal API, which persists them in Postgres. Demand for more frequent scanning outgrew the pipeline: sustained throughput was stuck around 10 scans/second against a target of at least 100/second — a 10x gap — and the constraint was to close it without adding hardware.
Two things were quietly throttling the whole pipeline. Checkers consumed Kafka messages essentially one at a time, even though most message types didn't actually need strict in-order processing beyond their own scan's correctness. And on the API side, persistence queries were holding connections from the client-side Postgres connection pool open longer than necessary; under real request volume the pool exhausted, and every consumer downstream ended up blocked waiting for a free connection — backpressure propagating all the way back through the fan-out.
Solution
Two changes, both aimed at the actual bottlenecks rather than at adding capacity. First, Kafka consumption moved from strictly serial, one-message-at-a-time processing to batched consumption, with each message in a batch handled concurrently in its own goroutine — parallelizing work that didn't need to be serialized in the first place. Second, the API layer's Postgres query patterns and connection-pool usage were tightened so persistence calls held a connection for a shorter, bounded window instead of monopolizing the pool, directly relieving the backpressure that had been rate-limiting every checker upstream.
Together, these pushed sustained throughput from ~10 scans/second to over 120 scans/second at peak scheduling — past the 10x target — with zero additional servers. The ceiling had been architectural (serialized consumption + connection contention), not physical.
What to steal
- Before reaching for more hardware, profile for "held resource, not held CPU" bottlenecks — a connection pool exhausted by queries that hold connections longer than necessary can throttle an entire fan-out pipeline as effectively as a CPU-bound stage would.
- Serial message processing is a default, not a requirement. Check whether your real ordering/correctness constraint is per-key (needs order within a key) or global (doesn't) — then batch and parallelize within whatever scope is actually safe.
- A 10x throughput win from software changes alone, with no new hardware, is a strong cost/performance story for a review: it's proof the ceiling was in the code, not the fleet.
- Fan-out pipelines (scheduler → queue → many workers → shared datastore) concentrate bottlenecks at the shared datastore and its client library — that's the first place to look when throughput plateaus, before the workers themselves.
Engineering Lens
This is a good example of a pillar-performance win that's really also a pillar-cost story — closing a 10x throughput gap without adding hardware is a directly quantifiable cost-avoidance argument, which is exactly the kind of framing that lands in a budget-constrained architecture review ("here's the throughput ceiling, here's why it was architectural, here's the fix that didn't require new spend"). The specific antipattern — a shared connection pool becoming the real bottleneck behind an apparently unrelated queue-consumer slowdown — is a pattern worth being able to recognize on sight; it shows up anywhere a fan-out pipeline terminates in a shared, stateful resource. It also has a genuine, non-forced analog to Mihir's network-tooling context: a scheduler polling or scanning a large device fleet (NetBox, Aegis) through a constrained queue-and-database backend is structurally the same shape, and the same two questions — "is this serialized when it doesn't need to be?" and "is something upstream holding a shared resource too long?" — apply directly.
Related
- Rate Limiting Algorithms
- Stripe: Graph Search and State Machines to Auto-Remediate a Global Database Fleet