Implementing Batch Processing Patterns
1. Batch Job Pattern
| Aspect | Detail |
|---|---|
| Definition | Long-running, finite job processing a bounded dataset |
| Triggers | Schedule, manual, event |
| Tools | Spring Batch, K8s Jobs, AWS Batch, Apache Beam |
| Properties | Restartable, observable, idempotent |
2. Scheduled Batch Pattern
| Mechanism | Detail |
|---|---|
| Cron | K8s CronJob, Quartz, Spring @Scheduled |
| Distributed Scheduler | Quartz cluster, Temporal cron workflows |
| Time Zone | Always UTC; document business TZ separately |
| Concurrency Policy | Forbid / Allow / Replace overlap |
3. Parallel Batch Processing Pattern
| Approach | Detail |
|---|---|
| Partition | Split input by key range / hash → workers |
| Multi-Threaded Step | Spring Batch parallel step |
| Distributed | Workers across nodes (Spark, Flink, Beam) |
| Aggregation | Combine partial results |
4. Chunking Pattern
| Aspect | Detail |
|---|---|
| Mechanism | Read N items → process → write as one unit (commit interval) |
| Benefits | Bounded memory; fewer commits; resumable |
| Tuning | Chunk size balances throughput vs commit overhead |
5. Checkpoint/Restart Pattern
| Aspect | Detail |
|---|---|
| Checkpoint | Persist progress (last record processed) |
| Restart | Resume from last checkpoint after failure |
| Storage | Job repository (Spring Batch), DB, S3 |
| Granularity | Per chunk; tradeoff between overhead and rework |
6. Master/Worker Pattern
| Role | Responsibility |
|---|---|
| Master | Splits work; tracks progress; handles failures |
| Worker | Processes one partition / task |
| Communication | Queue / direct RPC |
| Examples | Spring Batch remote partitioning, Spark driver/executors |
7. Staged Event-Driven Architecture Pattern
| Aspect | Detail |
|---|---|
| SEDA | App = pipeline of stages connected by queues |
| Each Stage | Independent thread pool; bounded queue |
| Benefit | Backpressure, isolation, observability per stage |
| Modern Forms | Reactive Streams, Kafka topic-per-stage pipelines |
8. Event-Driven Batch Pattern
| Trigger | Detail |
|---|---|
| File Arrival | S3 ObjectCreated event |
| Threshold | N messages accumulated |
| Time Window | Tumbling window completed |
| Use | Avoids cron lag; responds to data readiness |
9. Incremental Batch Pattern
| Aspect | Detail |
|---|---|
| Mechanism | Process only data since last successful run (high watermark) |
| Watermark | updated_at > last_run, or sequence number, or CDC offset |
| Benefit | Sublinear runtime; cheaper |
| Watch For | Late-arriving data; clock skew → use grace window |
10. Batch Compensation Pattern
| Aspect | Detail |
|---|---|
| Mechanism | Job to undo / correct prior batch's effects |
| Use Cases | Bad pricing run, miscalculated bills |
| Audit | Compensation logged; reversible |