Implementing Bulk Operations
1. Designing Bulk API Endpoints
| Aspect | Detail |
|---|---|
| Endpoint | POST /resources/bulk |
| Payload | Array of operations |
| Max items | Cap (e.g., 1000 per call) |
| Idempotency | Per-request key |
2. Implementing Bulk Insert Logic
Example: JDBC batch
jdbc.batchUpdate(
"INSERT INTO orders(id, customer_id, total) VALUES (?,?,?)",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
var o = orders.get(i);
ps.setObject(1, o.id());
ps.setObject(2, o.customerId());
ps.setLong(3, o.totalCents());
}
public int getBatchSize() { return orders.size(); }
});
3. Implementing Bulk Update Logic
Example: Set-based UPDATE
UPDATE orders SET status = 'archived'
WHERE created_at < now() - interval '180 days'
AND status = 'completed';
4. Implementing Bulk Delete Logic
| Practice | Detail |
|---|---|
| Chunked | DELETE in batches (e.g., 1k rows) |
| Soft delete first | Reversible |
| Confirmation | Two-step UI for destructive ops |
| Throttle | Avoid replication lag |
5. Implementing Bulk Import from CSV
Example: Streaming CSV reader
try (var reader = Files.newBufferedReader(path);
CSVParser p = CSVFormat.DEFAULT.builder().setHeader().build().parse(reader)) {
var batch = new ArrayList<Order>(1000);
for (var rec : p) {
batch.add(toOrder(rec));
if (batch.size() == 1000) { repo.saveAll(batch); batch.clear(); }
}
if (!batch.isEmpty()) repo.saveAll(batch);
}
6. Implementing Bulk Export
| Practice | Detail |
|---|---|
| Async | Background job for large sets |
| Stream | Don't materialize entire result in memory |
| Compress | gzip / zip output |
| Resume | Checkpoint cursor |
7. Handling Partial Failures
| Strategy | Detail |
|---|---|
| All-or-nothing | Single TX; one failure rolls back all |
| Best-effort | Per-item TX; continue on errors |
| Stop on first error | Useful for validation |
| Per-item result | Return success/failure per row |
8. Implementing Bulk Operation Progress
| Mechanism | Detail |
|---|---|
| Progress endpoint | GET /jobs/{id} returns percent |
| SSE stream | Push updates |
| Counters | processed / succeeded / failed |
| ETA | Based on throughput |
9. Implementing Bulk Operation Validation
| Phase | Detail |
|---|---|
| Pre-check | Validate all items before any side effect |
| Schema | Required fields, types |
| Business rules | Referential integrity |
| Per-item errors | Return list with line numbers |
10. Implementing Bulk Operation Rollback
| Approach | Detail |
|---|---|
| DB transaction | Single TX rollback |
| Compensation | Reverse each successful step |
| Saga | Distributed compensating steps |
| Snapshot | Restore from pre-op snapshot |
11. Implementing Bulk Operation Throttling
| Mechanism | Detail |
|---|---|
| Rate per tenant | Avoid noisy neighbor |
| Concurrency cap | Limit parallel bulk jobs |
| DB pause windows | Don't run during peak |
| Adaptive throttle | Slow down on replication lag |
12. Implementing Bulk Operation Reporting
| Output | Detail |
|---|---|
| Summary | Counts: ok, failed, skipped |
| Error CSV | Failed rows + reasons (downloadable) |
| Audit entry | Who, when, what |
| Notifications | Email on completion |