Developer/DataFlowPatterns/FanIn/Fundamentals/fan-in-aggregating-parallel-results-under-partial-failure

Fan-In: Aggregating Parallel Results Under Partial Failure

Concept

Fan-in is the converging half of fan-out/fan-in: multiple independent, concurrently-running operations produce results that need to be collected and combined into one output. Enterprise Integration Patterns formalizes this as Scatter-Gather (also called Broadcast-Aggregate) — a message is broadcast to multiple recipients (the scatter/fan-out half), and an Aggregator component collects the individual replies and combines them into a single response (the gather/fan-in half). The same shape appears without any messaging broker at all — a plain Promise.all/Task.WhenAll over N parallel calls is fan-in in miniature.

What makes fan-in a genuinely distinct engineering problem from fan-out isn't the collection mechanics — waiting for N futures to resolve is trivial — it's the completion rule: what does "done" mean when N independent, unreliable operations are in flight? The three real options are: wait for all N (fail the whole aggregate if any one fails or times out); wait for a quorum/first-K; or wait up to a fixed deadline and aggregate whatever has arrived by then, treating late or missing results as absent rather than failing the whole operation. Enterprise Integration Patterns describes two structural variants of the scatter side that affect this: a Recipient List scatter, where the sender controls and knows the exact list of recipients (so "all" has a concrete meaning), versus an auction-style scatter over a Publish-Subscribe Channel, where the sender doesn't know in advance how many subscribers exist or will respond at all — making a strict "wait for all" rule meaningless by construction.

Tradeoffs

Completion rule	Benefit	Cost
Wait for all N	Simplest correctness story — the aggregate is complete or it isn't	One slow or dead source blocks (or fails) the entire aggregate; brittle against partial availability
Quorum / first-K of N	Tolerates some sources being slow or down without failing the whole operation	Aggregate quality depends on which K happened to respond; needs a defined tie-break/quality rule
Deadline-bounded, partial-OK	Bounds worst-case latency regardless of how many sources are slow; graceful under partial outages	Callers must be designed to consume a "best-effort, possibly incomplete" result rather than assuming completeness

AWS's own guidance on scatter-gather makes the underlying point explicitly: a strong design "accepts that partial success is not always a failure" — if two of three sources respond and the third times out, continuing with a partial result is often the right call for both user experience and system resilience, as long as the completion rule was chosen deliberately rather than left as an implicit "wait forever" default.

When to use / when not to

Use whenever a single result genuinely needs input from more than one independent, concurrently-queryable source — price comparison across providers, fraud-check results from multiple independent scoring services, or rollup status from many fleet devices.
Especially important to design deliberately (not default to "wait for all") when the number of sources is large or their individual reliability is outside your control — the more sources feeding one aggregate, the higher the odds at least one is slow on any given run.
Don't reach for fan-in's aggregation machinery when there's really one authoritative source and the rest are best-effort enrichment — a simple optional call with its own timeout is enough; a formal Aggregator isn't needed for one nice-to-have.
Skip a bespoke Aggregator when the orchestration framework already provides one (Step Functions' Parallel state, Durable Functions' Task.WhenAll) — reach for a hand-rolled aggregator only when the completion rule is unusual enough that the framework default doesn't fit.

Common pitfall

Defaulting to "wait for all N to succeed" without ever deciding that on purpose — which turns every fan-in into a single point of fragility equal to the least reliable of its N sources. Collecting rollout status back from thousands of fleet devices, where a handful never respond, silently produces a stuck or perpetually-incomplete aggregate if "all" was never actually a deliberate choice. The fix isn't a specific completion rule — it's making sure one was chosen and tested against the "some sources never respond" case, not just the happy path where everything comes back on time.

Engineering Lens

The real design question in a fan-in isn't "how do I combine N results" — that's usually a one-line reduce. It's "what happens to the aggregate when fewer than N sources respond, and was that decided or discovered." A team that can point to a specific completion rule (all, quorum, deadline-bounded) and explain why it fits the business need has actually designed the aggregation; a team relying on the default behavior of whatever Promise.all-equivalent they reached for has implicitly chosen "wait for all, forever" without meaning to.