Developer/DataFlowPatterns/Pipeline/Fundamentals/pipeline-pattern-pipes-and-filters

Pipeline Pattern (Pipes and Filters)

Concept

A pipeline breaks a complex transformation into an ordered sequence of independent stages ("filters"), each of which consumes the output of the stage before it, does one well-defined unit of work, and hands its own output to the next stage over a connector ("pipe"). No stage knows anything about any other stage beyond the shape of the data crossing the pipe between them — a filter can be swapped, reordered relative to non-dependent stages, tested in isolation, or reused in a different pipeline without touching its neighbors. The pattern is old and general enough to show up under different names at every layer of the stack: Unix shell pipes (cat file.txt | grep pattern | sort | uniq), the map/reduce lineage of batch data processing, HTTP middleware chains, image/audio processing toolchains, and CI/CD build pipelines are all pipes-and-filters in different clothing.

Pipes can be implemented several ways depending on latency and durability needs: an in-process function composition or generator chain (cheapest, but stages share fate — one process crash loses the whole in-flight pipeline), OS pipes/sockets between separate processes (Unix shell model), or a durable message queue between stages (Kafka topics, SQS queues) when stages need to run on different machines, scale independently, or survive a downstream outage without losing in-flight data. The durable-queue variant is what turns a pipeline from a single-request processing chain into a horizontally scalable data pipeline: each stage can be scaled to its own bottleneck independently, and a stage that's temporarily down just causes its queue to back up rather than losing work.

Tradeoffs

Connector choice	Benefit	Cost
In-process (function/generator chain)	Lowest latency, no serialization overhead, simplest to write and debug	No independent scaling or failure isolation between stages — one stage's crash takes the whole pipeline down; can't distribute stages across machines
OS pipe / socket between processes	Stages can be separate processes/binaries (language-agnostic), some failure isolation	Still fate-shares with the machine; no built-in backpressure durability if a downstream process is slow or dead — data in flight is lost on a crash
Durable queue between stages (Kafka/SQS/etc.)	Stages scale and deploy independently; a stage outage backs up its queue instead of losing data; natural backpressure and replay	Highest latency and operational cost per stage transition (serialization, network hop, broker infrastructure); harder to reason about end-to-end ordering and exactly-once semantics across stage boundaries

A pipeline is not inherently parallel across records — depending on connector choice, later stages can start consuming a record as soon as an earlier stage emits it (streaming/pipelined execution, the way sort in a Unix pipe can begin reading before grep finishes writing everything), which overlaps stage latencies instead of summing them serially per record. But pipeline stages are still sequential and dependent for a single unit of data — for splitting one input in parallel across independent workers, see Scatter-Gather instead.

When to use / when not to

Use when a transformation naturally decomposes into an ordered sequence of independent steps with a clear data contract between each — image upload → validate → resize → store → generate thumbnail URL is a canonical example.
Especially valuable when different stages have different scaling profiles (e.g., a CPU-bound resize stage vs. a network-bound storage-upload stage) — durable-queue pipelines let each scale to its own bottleneck instead of over-provisioning the whole chain to match the slowest stage.
Don't reach for a multi-stage pipeline when the transformation is a single atomic step, or when stages have circular/bidirectional dependencies rather than a strict producer→consumer order — pipes-and-filters assumes one-directional flow.
Don't default to durable-queue connectors for latency-sensitive request/response paths unless independent stage scaling or fault isolation genuinely justifies the added hop latency — an in-process chain is usually the right starting point.

Common pitfall

Treating every stage's success as guaranteed and letting a mid-pipeline failure silently propagate bad or partial data downstream instead of stopping (or clearly flagging) the record at the point of failure. This is especially dangerous with durable-queue pipelines: a poison message that a stage can't process will either block that stage's consumer indefinitely (if retried forever) or get silently dropped (if the consumer swallows the exception and acks anyway) unless the pipeline has an explicit dead-letter path. Every stage boundary needs an explicit answer to "what happens to this record if this stage fails," not an assumption that failures won't happen.

Engineering Lens

The pipes-and-filters pattern's real design payoff isn't decomposition for its own sake — plenty of monolithic single-function transforms work fine — it's that each stage becomes independently testable, independently deployable, and independently scalable the moment the connector between two stages is a real interface (a queue, a socket, a well-typed function signature) rather than shared mutable state. The design review question worth asking isn't "did we break this into steps," it's "can stage 3 be redeployed, scaled, or replaced without stage 2 or stage 4 noticing or being redeployed too." A pipeline where stages share a database transaction or an in-memory object graph across the "pipe" boundary has the shape of pipes-and-filters without the actual benefit — it still fails and scales as one unit.