Developer/CommunicationPatterns/Sync-vs-Async/Fundamentals/choosing-between-api-queue-pubsub-streaming-and-polling

Choosing Between API, Queue, Pub/Sub, Event Streaming, and Polling

Concept

Every "how should these two components talk?" question collapses to the same handful of mechanisms, and the choice is not a matter of taste — it falls out of answering a short sequence of questions about the interaction's actual requirements. Picking the wrong one shows up later as either unnecessary latency coupling (sync where async would do) or unnecessary complexity and lost delivery guarantees (async/queue where a plain API call would do).

The decision tree, in order:

Does the caller need the answer within this same request/response cycle? → synchronous API call. If the UI or the calling service is blocked waiting for a value to act on right now, nothing async helps — it just adds a round trip.
Is the need ephemeral, single-purpose, and tolerant of no durability (a cache entry, a distributed lock, fan-out to whoever happens to be connected right now)? → use the in-memory store directly (e.g. Redis pub/sub, SETNX locks, INCR+TTL counters). There is no message to persist or retry — if nobody's listening, there's nothing to redeliver.
Does one event need exactly one worker to perform a job, with retry semantics? → task queue (a broker like RabbitMQ, or Redis used purely as a broker, fronted by a worker framework such as Celery). One producer, one logical consumer group, at-least-once delivery, acknowledgments.
Do many independent systems need to react to the same event, possibly at different times, possibly needing to replay history? → event stream (Kafka, Redpanda, or similar log-based systems). Multiple consumer groups each track their own offset into the same durable log; a new consumer can be added later and replay everything from the beginning.
Is there no real-time channel available, and is the check infrequent or low-stakes? → polling, as a fallback rather than a first choice. It trades implementation simplicity for wasted requests and added latency between the actual event and the caller noticing it.

A mechanism tied to a specific language runtime (e.g. a queue library that only has a client for one language) never enters this decision at the architecture layer — it gets ruled out at the implementation layer, by what your services are written in, before the architectural question of "queue vs stream vs API" is even asked.

Tradeoffs

Mechanism	Delivery model	Durability / replay	Cost
Sync API	Request blocks for response	None — no message to redeliver	Couples caller's latency to callee's; simplest mental model
In-memory store (Redis pub/sub, locks, counters)	Fire-and-forget to current subscribers	None by design — ephemeral	Fastest, cheapest; wrong choice if the message must survive no-one listening
Task queue (RabbitMQ/Celery)	One event → one worker, ack/retry	Durable until acked; no replay after consumption	Needs a broker + worker infra; strong guarantees for a single consumer role
Event stream (Kafka/Redpanda)	One event → many independent consumer groups	Durable log, replayable by offset	Highest operational weight (partitions, consumer group management, storage growth); justified only when replay or multi-consumer fan-out is a real requirement
Polling	Caller repeatedly asks	N/A — caller controls when it checks	Cheapest to build, worst latency and resource efficiency at scale

The throughline: each step up this list (API → in-memory → queue → stream) buys stronger guarantees (durability, retry, replay, independent multi-consumer fan-out) at the cost of more infrastructure to run and reason about. The discipline is choosing the cheapest mechanism that actually satisfies the requirement, not defaulting to the most powerful one "in case we need it later."

When to use / when not to

Reach for a sync API whenever the caller is a human or a system that cannot proceed without the answer — search-as-you-type, a payment charge, a page render.
Reach for the in-memory store when the data is disposable by nature: caches, locks, rate-limit counters, WebSocket fan-out where persistence is handled elsewhere (e.g. the message is also written to a database of record).
Reach for a task queue when there is one job and one kind of worker that must do it reliably, off the request path, with retries — sending an email, generating a thumbnail, reindexing a document.
Reach for an event stream only once there are genuinely multiple, independent, possibly-added-later consumers of the same event, or a real need to reprocess history (backfilling a new analytics pipeline, replaying events into a new ML model). Don't reach for it "because Kafka is what big companies use" — it is the deliberately heavier option in this tree, not the default.
Reach for polling only when no push channel exists (a third-party API with no webhooks) or the check is infrequent enough that the wasted requests don't matter. If check frequency would be high or latency matters, prefer a push mechanism (WebSocket, SSE) over increasing poll frequency.
Anything that must remain correct under concurrent access with financial or state-machine consequences (e.g. "has this slot already been booked?") belongs in the system of record's own transactions (database row locks), not in a queue or a Redis lock — queues and caches manage side effects of a decision, they are not where the decision's correctness should live.

Common pitfall

Treating this as a single global choice ("we use Kafka" or "we use REST") instead of an interaction-by-interaction decision. Real systems use all five mechanisms simultaneously for different edges of the same feature: a synchronous API call for the part the user is waiting on, a task queue for the side effects that follow it, an in-memory store for ephemeral fan-out, and — only if a genuine multi-consumer/replay need exists — an event stream for cross-cutting concerns like analytics. Picking one mechanism as "the" architecture and forcing every interaction through it (e.g. routing a synchronous payment confirmation through an async queue, or building a queue-based worker for something that only ever polls a third-party API) is what actually produces the "unnecessary latency coupling or unnecessary complexity" this topic exists to avoid.

Engineering Lens

The senior-to-staff jump in this area is recognizing that "which message system should we use" is usually the wrong question — the right one is "what does this specific interaction need: an answer now, at-least-once delivery to one worker, or replayable fan-out to many?" Each answer names its own mechanism; there is rarely one correct answer for an entire system. The failure mode to watch for in review is over-provisioning guarantees the interaction doesn't need (introducing an event stream for what is really a single background job) or under-provisioning them (polling for something that needed a push channel, or firing a queue message for something the caller was actually blocked waiting on). The tree above is a fast, repeatable way to make that call per-interaction instead of re-litigating "queue vs stream vs API" from scratch every time a new feature touches inter-service communication.

Sources

Discussion synthesizing Celery/RabbitMQ vs Kafka vs Redis vs polling tradeoffs for a marketplace platform (2026-08-27)