Implementing Background Job Processing
1. Designing Job Queue System
| Component | Detail |
|---|---|
| Queue | Pending jobs (Redis, DB, broker) |
| Worker pool | Polls and executes |
| Result store | Status + output |
| Scheduler | For delayed/recurring jobs |
2. Implementing Scheduled Jobs
Example: @Scheduled cron
@Configuration @EnableScheduling
public class ScheduleConfig {}
@Service
public class ReportJob {
@Scheduled(cron = "0 0 2 * * *", zone = "UTC") // 02:00 UTC daily
public void daily() { reports.generateDaily(); }
}3. Implementing Cron Job Processor
| Field | Range |
|---|---|
| second | 0-59 (Spring includes seconds) |
| minute | 0-59 |
| hour | 0-23 |
| day-of-month | 1-31 |
| month | 1-12 |
| day-of-week | 0-7 (0 or 7 = Sun) |
4. Implementing Job Scheduling Logic
| Library | Detail |
|---|---|
| Spring @Scheduled | Simple, in-process |
| Quartz | JDBC-backed, clustered |
| ShedLock | Distributed lock for @Scheduled |
| db-scheduler | Lightweight DB-backed |
5. Handling Job Failures
| Strategy | Detail |
|---|---|
| Retry | With backoff |
| DLQ | Manual investigation |
| Alerting | On repeated failure |
| Idempotency | Required for safe retry |
6. Implementing Job Retry Logic
| Setting | Recommendation |
|---|---|
| Max attempts | 3-5 |
| Backoff | Exponential + jitter |
| Per-error policy | Don't retry 4xx-style failures |
| Visible in UI | Show attempt count |
7. Implementing Distributed Job Locks
Example: ShedLock
@Scheduled(cron = "0 */5 * * * *")
@SchedulerLock(name = "syncFx", lockAtMostFor = "PT4M", lockAtLeastFor = "PT30S")
public void syncFx() { fxService.sync(); }
// Only one node executes at a time8. Implementing Job Priority Queues
| Approach | Detail |
|---|---|
| Multiple queues | high / default / low; workers poll high first |
| Score-based | Redis sorted set by priority |
| Starvation guard | Periodically promote low |
9. Implementing Job Status Tracking
| Status | Meaning |
|---|---|
| PENDING | Queued |
| RUNNING | Worker executing |
| SUCCEEDED | Completed OK |
| FAILED | Permanent failure |
| RETRYING | Will be re-attempted |
| CANCELLED | User-cancelled |
10. Implementing Long-Running Job Logic
| Practice | Detail |
|---|---|
| Checkpointing | Save progress periodically |
| Resumable | Pick up where it left off |
| Heartbeat | Detect stalled workers |
| Cancellation | Cooperative interruption |
| Progress reporting | Percent / step counters |
11. Handling Job Concurrency
| Setting | Detail |
|---|---|
| Worker count | Per node |
| Per-queue concurrency | Isolate noisy neighbors |
| Per-tenant cap | Fairness |
| Backpressure | Pause polling when overloaded |
12. Implementing Job Result Storage
| Storage | Use |
|---|---|
| DB row | Status + small result |
| Object store (S3) | Large outputs |
| Redis with TTL | Short-lived results |
| Streamed | For real-time consumers |