Designing Asynchronous Processing
1. Designing Background Job Architecture
| Component | Detail |
|---|---|
| Producer | Web/API enqueues jobs |
| Queue | Persistent (Redis, SQS, RabbitMQ, DB table) |
| Worker pool | Long-running consumers |
| Orchestrator | Optional: dependencies, schedules |
| Status store | Job status / result |
2. Designing Job Queue Systems
| System | Strength |
|---|---|
| Sidekiq | Ruby; Redis-backed |
| Celery | Python; broker-agnostic |
| BullMQ | Node.js; Redis |
| RQ | Simple Python jobs |
| Sidekiq Enterprise / Resque | Production-grade |
| DB-backed (Que, Solid Queue) | Single-store ops |
3. Designing Long-Running Task Architecture
| Pattern | Detail |
|---|---|
| Async + status endpoint | POST returns 202 + job_id; GET /jobs/:id |
| Webhook callback | Notify URL on completion |
| SSE / WebSocket progress | Push updates to client |
| Result storage | S3 / DB; signed URL to result |
| Workflow engine | Temporal for multi-day jobs |
4. Designing Retry Mechanisms with Backoff
| Strategy | Detail |
|---|---|
| Exponential | 1s, 2s, 4s, 8s ... |
| Jitter | ± random offset to spread load |
| Cap | Max delay (e.g., 5 min) |
| Max attempts | 5–25 depending on criticality |
| Permanent vs transient | Don't retry 4xx |
5. Designing Scheduled Jobs Architecture
| Tool | Detail |
|---|---|
| cron / systemd timer | Single host; SPOF |
| K8s CronJob | Cluster-managed |
| Quartz / sidekiq-cron | App-level scheduler |
| Cloud schedulers | EventBridge, Cloud Scheduler |
| Distributed leader | Ensures single execution |
6. Designing Task Priority Queues
| Approach | Detail |
|---|---|
| Multiple queues | critical, high, default, low |
| Workers per priority | Pool dedicated |
| Weighted polling | Avoid low-prio starvation |
| SLA-driven | Critical: 1s p99; low: minutes |
7. Designing Worker Pool Sizing Strategy
| Workload | Sizing |
|---|---|
| CPU-bound | ≈ # CPU cores |
| I/O-bound | 10–100× cores |
| Mixed | Tune via load tests |
| Auto-scale | By queue depth or job age |
| Per-tenant cap | Prevent noisy neighbor |
8. Designing Job Status Tracking System
| Field | Detail |
|---|---|
| job_id | UUID |
| status | queued / running / done / failed / canceled |
| progress | 0–100 or step name |
| started_at / finished_at | Timestamps |
| attempts | Retry count |
| result_url / error | Output |
9. Designing Job Failure and Compensation
| Action | Detail |
|---|---|
| Auto-retry | Transient errors |
| DLQ + alert | Permanent failures |
| Compensation hook | Cleanup partial work |
| Manual replay | Admin tool to re-enqueue |
10. Designing Batch Processing Architecture
| Element | Detail |
|---|---|
| Chunking | Process in fixed-size chunks |
| Checkpointing | Resume from last chunk |
| Parallelism | Partition input across workers |
| Tools | Spring Batch, Apache Beam, Airflow + tasks |
11. Designing Job Deduplication
| Strategy | Detail |
|---|---|
| Unique key | (type, business_id) prevents enqueue |
| Lock during execution | Single worker per key |
| SETNX TTL | Redis-based dedup |
| DB constraint | Unique index on job table |
12. Designing Job Orchestration
| Tool | Use Case |
|---|---|
| Airflow | DAG-based ETL |
| Prefect | Pythonic flows |
| Dagster | Asset-oriented data pipelines |
| Temporal | Long-running stateful workflows |
| Step Functions | AWS-native serverless |