Design a Distributed Job Scheduler
Scenario prompt
Design a cron-as-a-service platform (Airflow/Temporal-style) that lets teams across an org register recurring and one-off jobs. Requirements:
- A scheduled trigger must fire exactly once, even though the scheduler itself runs as multiple replicas for HA
- A worker crashing mid-job must not lose the job — but also must not silently duplicate its side effects (double-charging a customer, double-sending an email)
- Must scale to tens of thousands of scheduled jobs with wildly different cadences (every minute to once a year) without a single node having to scan the whole job table on every tick
- Needs retries with backoff for transient failures, without blindly retrying jobs that aren't safe to run twice
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd stop two scheduler replicas from double-firing the same trigger, and how you'd decide which failed jobs are safe to retry.
Model solution
Separate "deciding a job is due" from "executing it" — the scheduler enqueues, it never runs the job itself. The scheduling tier's only job is to notice a trigger is due and place a message on a durable queue; a separate pool of workers dequeues and executes. This decoupling means a scheduler replica crashing mid-decision doesn't lose in-flight execution state, and worker capacity can scale independently of trigger-evaluation capacity.
Prevent double-firing with a lease, not a hope. With multiple scheduler replicas polling the same job table, two replicas can otherwise both decide the same trigger is due in the same tick. The fix is a short-lived lease/lock per job (a DB row-level SELECT ... FOR UPDATE SKIP LOCKED, or a distributed lock via etcd/ZooKeeper) — whichever replica acquires the lease owns firing that trigger this cycle. This is the same leader-election-for-a-slice-of-work pattern that shows up anywhere multiple instances share ownership of a partitioned resource.
Avoid the hot-scan problem with time-bucketed sharding. Scanning "all jobs due in the next minute" against a flat table of tens of thousands of rows on every tick doesn't scale. Bucketing jobs by their next-fire time (a min-heap or a sharded index keyed by next_run_at) and sharding job ownership across scheduler instances by consistent hashing on job ID keeps each tick's work bounded to what's actually due, similar in spirit to Database Sharding Strategies.
Idempotency keys turn "at-least-once delivery" into "effectively-once execution." True exactly-once execution across a network is not achievable, so the practical target is at-least-once delivery from the queue plus an idempotency key per scheduled execution (job ID + scheduled timestamp) that the job handler checks before applying side effects — exactly the mechanism in Idempotency Keys. A worker crashing after doing the work but before acking the queue message causes a redelivery; the idempotency check is what stops that redelivery from re-charging a customer.
Retries have to know whether a job is safe to repeat. Jobs get classified at registration time as idempotent or not. Idempotent jobs retry automatically with exponential backoff up to a cap, then land in a dead-letter queue. Non-idempotent jobs that fail get surfaced for manual review rather than auto-retried — retrying blind is how a transient network blip turns into a double-executed side effect.
Gaps to revisit
- Long-running jobs whose execution time exceeds the schedule interval — does the next trigger skip, queue up, or run concurrently, and who decides per-job?
- Clock skew and DST/timezone handling for cron-style schedules defined in local time across regions
- Rebalancing job shards across scheduler replicas during a scaling event without missing a trigger that falls in the gap
Engineering Lens
This challenge is really three composed problems wearing a "cron" costume: leader election over partitioned work, at-least-once delivery, and idempotent side effects. Recognizing that composition — rather than treating "build a scheduler" as its own bespoke problem — is what lets the same mental model transfer to a payments settlement batch, a nightly data pipeline, or a certificate-rotation job. In Fintech/Capital Markets specifically, this exact design question has real teeth: an EOD reconciliation or settlement job that fires twice because of a scheduler bug can mean a duplicated wire transfer, which is precisely why idempotency keys aren't an optional nicety here — they're the difference between a retry and an incident.