Designing Batch Processing Systems
1. Designing ETL Pipeline Architecture
| Stage | Detail |
|---|---|
| Extract | Sources: DB, files, APIs |
| Transform | Cleanse, normalize, enrich |
| Load | Target: DW, lake, search |
| Orchestrator | Airflow, Dagster, Prefect |
| Compute | Spark, dbt, Beam |
2. Designing Data Processing Workflows
| Element | Detail |
|---|---|
| DAG | Directed acyclic dependencies |
| Task | Atomic unit; idempotent |
| Trigger | Schedule, sensor, manual |
| SLA | Per-task deadline; alert on miss |
3. Designing Map-Reduce Patterns
| Phase | Detail |
|---|---|
| Map | Parallel transformation per record |
| Shuffle | Partition + sort by key |
| Reduce | Aggregate per key |
| Combiner | Local pre-reduce; reduce shuffle |
| Modern | Spark RDD/DF, Beam, Flink batch |
4. Designing Batch Job Scheduling
| Mechanism | Detail |
|---|---|
| Cron expression | Time-based |
| Sensor | Wait for file/event |
| Backfill | Run for past intervals |
| Catchup | Disable to skip missed schedules |
5. Designing Data Validation and Cleansing
| Tool / Check | Detail |
|---|---|
| Great Expectations | Declarative data tests |
| dbt tests | not_null, unique, accepted_values, relationships |
| Schema validation | JSON Schema, Avro |
| Anomaly detection | Volume / distribution checks |
| Quarantine | Bad rows → side table |
6. Designing Idempotent Batch Jobs
| Technique | Detail |
|---|---|
| Partition overwrite | Re-run replaces partition |
| MERGE / UPSERT | Same input → same output |
| Job-run id in output | Detect/clean prior partial runs |
| Atomic publish | Write to temp, swap in |
7. Designing Batch Job Monitoring
| Metric | Detail |
|---|---|
| Duration | Trend over time; alert on regression |
| Rows processed | Output volume sanity |
| Success rate | Per-task |
| Cost | Cluster hours / DBUs |
| Lineage | Upstream/downstream impact |
8. Designing Incremental Processing
| Approach | Detail |
|---|---|
| Watermark column | Process WHERE updated_at > last_run |
| CDC | Only changed rows |
| Change feeds | Iceberg/Delta/Hudi changelog |
| Snapshot diff | Compare with previous full |
9. Designing Batch Job Failure Handling
| Action | Detail |
|---|---|
| Retry with backoff | Transient infra issues |
| Skip + alert | Bad input row; quarantine |
| Resume from checkpoint | Don't restart whole pipeline |
| Manual rerun | Backfill specific partition |
10. Designing Distributed Batch Processing
| Engine | Strength |
|---|---|
| Apache Spark | General compute, ML |
| Apache Beam | Unified batch + stream |
| Trino / Presto | Federated SQL |
| Dask | Python-native |
| Snowflake / BigQuery | Managed analytic SQL |
11. Designing Batch Job Checkpointing
| Technique | Detail |
|---|---|
| Per-stage outputs | Persist intermediates to storage |
| Spark checkpoint() | Truncate lineage, recover faster |
| Workflow markers | Track completed steps |
| Idempotent restart | Skip done steps |
12. Designing Batch Job Parallelization
| Strategy | Detail |
|---|---|
| Data partitioning | By date / shard / hash |
| Right-size partitions | ~128MB–1GB each |
| Avoid skew | Salt heavy keys |
| Adaptive query (Spark AQE) | Auto-rebalance |
| Resource sizing | Memory/CPU per executor |