Developer/Compute/BatchProcessing/Fundamentals/idempotent-batch-job-design

Idempotent Batch Job Design

Concept

A batch job runs unattended, on a schedule or a trigger, and nobody is watching it live to catch a partial failure the moment it happens. That combination — unattended execution plus the near-certainty that something eventually fails partway through a long-running job (a network blip, an OOM kill, a deploy that restarts the worker mid-run) — means the real design question for a batch job isn't "will it fail," it's "what happens when it's re-run after failing partway." A job whose answer to that is "the same result, cleanly" is idempotent; a job whose answer is "duplicate rows, double-charged payouts, or corrupted aggregates" is not, and that gap is invisible in testing because a clean single run looks identical either way.

Idempotency in a batch context is achieved by a small number of concrete techniques rather than a vague design principle:

Deterministic keys + upsert. Each unit of output gets a key derived from its input (a natural key, or a hash of the input row), and the write is an upsert (INSERT ... ON CONFLICT DO UPDATE) or a full overwrite of that key's row, never a blind INSERT. Re-running the same input twice converges to the same row instead of producing a duplicate.
Partition/window overwrite instead of append. For aggregation jobs (nightly rollups, report generation), write each run's output to a partition keyed by the batch window (e.g. date=2026-08-27) and overwrite that partition wholesale, rather than appending new rows to an ever-growing table. A re-run of the same day's job replaces that day's data instead of adding a second copy of it.
A processed-marker, checked before work starts. The job records, transactionally with its own output, that batch window X completed. On start, it checks that marker first — a job that's already fully completed for that window exits as a no-op rather than redoing (and potentially re-emitting side effects like an email or a payout) work that already succeeded.
Single-instance execution via a distributed lock. A batch job that overruns its schedule (a job scheduled every 10 minutes that takes 30) will overlap with the next scheduled run unless something stops it — a Redis lock (SET key val NX EX <2x max job duration>) or a lighter-weight local flock for single-host cron both work: the second invocation checks the lock, sees the first still running, and exits immediately instead of starting a concurrent second copy against the same data.

Tradeoffs

Approach	Benefit	Cost
Blind append, no lock	Simplest to write	Any retry or overlap duplicates rows; failure recovery means manual cleanup
Deterministic key + upsert	Retry-safe by construction, no dedup step needed downstream	Requires a real key for every unit of work; not every domain has one to derive
Partition overwrite	Simple mental model ("this run replaces this window's data")	Only fits aggregation-shaped jobs where the whole window's output is naturally coherent
Distributed lock (Redis/ZooKeeper/etcd)	Works across multiple hosts/workers, not just one machine	Extra infrastructure dependency; a stale lock (crashed holder, no TTL) can wedge the job indefinitely if the TTL is set wrong
Local lock (`flock`)	Zero extra infrastructure, one lock file	Only works if the job always runs on the same single host — doesn't generalize to a horizontally scaled worker fleet

The lock-TTL tradeoff deserves its own note: too short and a legitimately-still-running job gets treated as dead, and a second copy starts alongside it — the exact failure the lock exists to prevent. Too long, and a real crash leaves the lock held long after the job actually died, blocking every subsequent scheduled run until the TTL expires or someone intervenes by hand.

When to use / when not to

Use deterministic-key upserts and window-partition overwrites for anything that runs on a schedule and isn't purely additive — nightly aggregation, payout calculation, report generation, data warehouse loads.
Add a distributed lock the moment a batch job runs on more than one host, or the moment its runtime can plausibly exceed its own schedule interval under load — both conditions make an unprotected overlap a "when," not an "if."
Skip the lock/marker machinery for jobs that are naturally, trivially idempotent already (e.g. a job that only reads and never writes, or one that always produces the exact same deterministic output regardless of how many times it runs on the exact same input) — added machinery with no failure mode it protects against is just cost.
Don't rely on "it finished quickly in testing so overlap won't happen in production" — the failure mode is specifically a slow run under real load or real data volume, which is exactly the condition testing rarely reproduces.

Common pitfall

Treating "the job succeeded" and "the job's side effects happened exactly once" as the same fact. A job can crash after writing its output rows but before recording its own completion marker; the next scheduled run, seeing no marker, redoes the entire window — including any non-idempotent side effect embedded in the job, like sending a payout-confirmation email or calling a third-party API. The fix is ordering: durable output write and completion marker must be part of the same atomic unit (same transaction, or the marker write is the very last step and is itself checked defensively before any external side effect fires), not two separate steps that can be torn apart by a crash landing between them.

Engineering Lens

Batch idempotency is the same distributed-systems problem webhook delivery and message-queue consumption both have — a network or a process can fail between "I did the work" and "I recorded that I did the work," and no amount of careful sequencing removes that gap, only shrinks and controls it. The pattern that recurs everywhere this problem shows up is the same: make the unit of retry small and deterministic (a key, a window, a message ID), and make "did I already do this" a cheap, correct check rather than an assumption. A batch pipeline that has internalized this treats every scheduled run as a possible re-run of a previous partial failure by default, the same reflex applied to retried webhooks or retried queue messages — because it's structurally the same failure mode wearing a different schedule.