Scheduled Jobs & Data Reconciliation
The 3-Line Engineering Rule
Never run bare in-memory `@Scheduled` cron jobs across multi-instance deployments; always protect them with distributed locks and build shadow reconciliation loops.
If your app runs 5 replicas, a bare cron job fires 5 times concurrently, sending duplicate billing invoices, charging customers 5x, or producing race conditions.
Use ShedLock or Redis distributed locks with safety hold times, make jobs chunked and idempotent, and build daily reconciliation loops comparing system balances.
Architectural Blueprint & Sequence
Hover over nodes to inspect state transitionsApp Replica 1
Cron triggers at 02:00
App Replica 2
Cron triggers at 02:00
Distributed Lock Table
shedlock table in DB / Redis
Node 1 Wins Lock
Executes reconciliation chunk
Node 2 Skips
Already locked -> Exit cleanly
Reconciliation Ledger
Flag discrepancies > 0
Key Engineering Takeaways
- Use ShedLock or Redis distributed locks to guarantee single-instance execution.
- Configure `lockAtLeastFor` to prevent clock drift from triggering duplicate executions.
- Configure `lockAtMostFor` so a crashed worker doesn't freeze the job indefinitely.
- Always design reconciliation jobs as 'catch-up' loops comparing internal DB state with external provider state (e.g. Stripe/PayPal).
The Problem & Production Reality
In modern cloud deployments, services run across multiple pods or auto-scaled containers. A standard cron annotation (`@Scheduled(cron = "0 0 2 * * ?")`) will fire simultaneously on every running pod. Furthermore, external network glitches cause occasional discrepancies between your database and payment providers; an automated reconciliation job detects and repairs these inconsistencies.
Anti-Pattern: Unsynchronized Multi-Pod Scheduled Runs
Running scheduled jobs in clustered environments without distributed locking triggers race conditions and duplicate emails/charges.
// ❌ ANTI-PATTERN: Fires on all 5 pods at 02:00 AM!
@Scheduled(cron = "0 0 2 * * ?")
public void runDailyBilling() {
// 5 pods execute this at the exact same second!
List<Subscription> subs = subRepo.findDue();
for (Subscription s : subs) {
billingGateway.charge(s); // Customers charged 5 times!
}
}Production Pattern: ShedLock Distributed Coordination & Chunked Processing
Enforce single-replica execution via ShedLock, set safety lock boundaries, and process records in idempotent chunks.
// ✅ PRODUCTION PATTERN: ShedLock + Batch Idempotent Reconciliation
@Component
public class DailyReconciliationJob {
private final SubscriptionRepository subRepo;
private final PaymentGatewayClient gatewayClient;
@Scheduled(cron = "0 0 2 * * ?")
@SchedulerLock(
name = "daily_subscription_billing",
lockAtLeastFor = "5m", // Keep lock even if job finishes fast (clock drift guard)
lockAtMostFor = "30m" // Auto-release if node dies
)
public void runReconciliation() {
log.info("Acquired distributed lock. Starting daily reconciliation loop.");
int page = 0;
Page<Subscription> batch;
do {
batch = subRepo.findByStatusAndNextBillingDateBefore(
SubscriptionStatus.ACTIVE, LocalDate.now(), PageRequest.of(page, 100)
);
for (Subscription sub : batch) {
reconcileSingleSubscription(sub);
}
page++;
} while (batch.hasNext());
}
private void reconcileSingleSubscription(Subscription sub) {
// Idempotent reconciliation logic: check remote provider state first
GatewayStatus remote = gatewayClient.getSubscriptionStatus(sub.getGatewaySubId());
if (remote.isPaid() && !sub.isMarkedPaid()) {
sub.markPaid();
subRepo.save(sub);
}
}
}Trade-off Analysis & Staff Recommendations
| Aspect | Pros / Strengths | Cons / Hazards | Staff Recommendation |
|---|---|---|---|
| Database-backed ShedLock vs Redis Redlock | Relational DB ShedLock uses your existing PostgreSQL/MySQL database without new infrastructure. | Puts minor lock acquisition queries on the relational DB. | Use the relational DB ShedLock table unless your scheduled jobs run every few seconds. |
| Full Scan vs Chunked Keyset Pagination | Chunked pagination keeps memory usage constant regardless of table size. | Requires ordering by immutable auto-incrementing ID. | Never load 100k rows into memory (`findAll()`); process in batches of 500. |