Design a Distributed Job Scheduler

Distributed job schedulers coordinate execution of millions of recurring (cron) and ad-hoc delayed tasks across worker clusters with at-least-once execution guarantees, dependency graphs (DAGs), and automatic retry semantics.

1. System Requirements

Functional Requirements

Submit Jobs: Users/services can submit one-time delayed jobs (run_at: timestamp) or recurring cron jobs (cron: "*/5 * * * *").
Execution Guarantees: Tasks must trigger within a few seconds of scheduled execution time.
Task Dependencies (DAGs): Support execution pipelines where Task B only runs after Task A completes successfully.
Retry & Backoff: Automatic retries with exponential backoff and jitter on worker failure.
Job Monitoring & Status: Real-time dashboard to inspect job execution state (PENDING, RUNNING, SUCCESS, FAILED).

Non-Functional Requirements

Scalability: Support scheduling 100M+ tasks per day with peak trigger rates of 50,000 tasks/second.
High Availability: Master/Coordinator nodes must failover seamlessly without dropping scheduled tasks.
Durability: Zero task loss if a node crashes right before trigger time.

2. Core Architecture: Hierarchical Time Wheel & Database Partitioning

The Hierarchical Hashed Timing Wheel

Instead of polling the database with SELECT * FROM jobs WHERE trigger_time <= NOW() (which locks tables and scales poorly), the scheduler uses an in-memory Hierarchical Timing Wheel:

Divided into Second, Minute, Hour, and Day circular ring buffers.
Tasks are placed into buckets corresponding to execution offsets.
As the clock advances each second, the current bucket's tasks are immediately drained into a distributed task queue (Kafka / RabbitMQ).

3. High-Level Architecture Blueprint

code

                                  +-----------------------+
                                  |    Client Services    |
                                  +-----------+-----------+
                                              |
                                              | POST /jobs
                                              v
                                  +-----------------------+
                                  |     Scheduler API     |
                                  +-----------+-----------+
                                              |
                                              v
                                  +-----------------------+
                                  | Primary Storage DB    |
                                  | (PostgreSQL / TiDB)   |
                                  +-----+-----------+-----+
                                        |           |
             +--------------------------+           +--------------------------+
             |                                                                 |
             v                                                                 v
+------------------------+                                         +------------------------+
| Scheduler Coordinator  |                                         |  Distributed Message   |
| (Leader Election via   |                                         |  Queue (Kafka / SQS)   |
| ZooKeeper / Raft)      |                                         +-----------+------------+
+-----------+------------+                                                     |
            |                                                                  v
            +==== Dispatches Tasks Due Now ====================================>+
                                                                               |
                                                                               v
                                                                   +------------------------+
                                                                   | Worker Nodes Cluster   |
                                                                   | (Task Execution Pods)  |
                                                                   +------------------------+

4. Staff-Level Deep Dives

Deep Dive 1: Preventing Duplicate Executions

In distributed systems, networks partition and workers experience garbage collection pauses.

Distributed Lock with Fencing Tokens: Before executing, the worker acquires a Redis / ZooKeeper lock. The lock issues a monotonically incrementing fencing token. If a paused worker wakes up, its stale token is rejected by the database.
Idempotency Keys: Workers pass the task_execution_id to downstream APIs to ensure side effects are safely deduplicated.

Key Architecture Trade-Offs & Decision Matrix

Dimension	Option A: Asynchronous / Decoupled	Option B: Synchronous RPC	Trade-off & Recommendation
Communication Pattern	Message Queues (Kafka / SQS)	Direct gRPC / REST Calls	Message queues prevent cascading failures and provide backpressure buffering.
Failure Handling	Dead Letter Queue (DLQ) with Exponential Backoff	Immediate Circuit Breaker trip	DLQ preserves unparseable messages for manual operational inspection.
Ordering Guarantees	Partition-level Strict FIFO	Unordered Parallel Execution	Partition-key routing guarantees chronological order per entity without global lock contention.

Design a Distributed Job Scheduler

Design a Distributed Job Scheduler

1. System Requirements

Functional Requirements

Non-Functional Requirements

2. Core Architecture: Hierarchical Time Wheel & Database Partitioning

The Hierarchical Hashed Timing Wheel

3. High-Level Architecture Blueprint

4. Staff-Level Deep Dives

Deep Dive 1: Preventing Duplicate Executions

Key Architecture Trade-Offs & Decision Matrix

Related Topics

Consistent Hashing Architecture & Implementation

Event-Driven Architecture & Scaling with Apache Kafka

Dealing with High Concurrency & Contention

Cheat Sheet Utility