Managing Long-Running Background Tasks

Operations like generating complex tax PDF reports, running ML model inference, or exporting database backups take seconds to minutes, making synchronous HTTP request/response loops unfeasible.

1. Asynchronous Job Lifecycle Pattern

1. Submit: Client sends POST /reports/generate.
2. Accept (202 Accepted): Server enqueues job into Kafka / SQS and returns HTTP 202 Accepted with a Location: /reports/status/{job_id} header.
3. Execution: Celery / Temporal worker consumes the job, updating status in Redis.
4. Completion Delivery:
5. Client Polling: Client polls status endpoint every 2 seconds with exponential backoff.
6. Server-Sent Events (SSE): Server pushes real-time progress percentages (progress: 75%).
7. Webhooks: For B2B integrations, server executes an HTTP POST to the client's webhook endpoint upon job completion.

Key Architecture Trade-Offs & Decision Matrix