Implementing Bulk Data Operations
1. Designing Bulk Import Endpoints
| Endpoint Pattern | Use |
|---|---|
POST /users/import | Sync small imports |
POST /jobs/import + 202 Accepted | Async large imports |
POST /imports as resource | Trackable import job (CRUD on imports) |
2. Designing Bulk Export Endpoints
| Endpoint | Behavior |
|---|---|
GET /users/export?format=csv | Sync stream |
POST /exports + poll | Async; returns download URL when ready |
| Filtered export | Apply same filters as list endpoint |
3. Implementing CSV/JSON Import
| Format | Content-Type | Notes |
|---|---|---|
| CSV | text/csv | Header row required; UTF-8 BOM optional |
| NDJSON | application/x-ndjson | One JSON object per line; streamable |
| JSON Lines | application/jsonl | Same as NDJSON |
| JSON array | application/json | Must fit in memory |
4. Implementing CSV/JSON Export
| Header | Value |
|---|---|
Content-Type | text/csv; charset=utf-8 |
Content-Disposition | attachment; filename="users-2026-05-15.csv" |
Transfer-Encoding | chunked (streaming) |
5. Handling Large Dataset Processing
| Technique | Benefit |
|---|---|
| Streaming parser | Constant memory |
| Chunked processing (1000-row batches) | Reduced DB round-trips |
| Background queue (SQS, RabbitMQ) | Decouple from request |
| Worker scaling | Horizontal throughput |
6. Implementing Background Processing
Example: Async Job Pattern
POST /imports
→ 202 Accepted
Location: /imports/abc-123
{"id": "abc-123", "status": "queued"}
GET /imports/abc-123
→ 200 OK
{"id": "abc-123", "status": "processing", "progress": 0.42, "processedRows": 4200}
GET /imports/abc-123 (later)
→ 200 OK
{"id": "abc-123", "status": "completed", "result": {"created": 9800, "failed": 200}}7. Providing Progress Indicators
| Field | Type |
|---|---|
status | queued | processing | completed | failed |
progress | 0.0 - 1.0 (fraction) |
processedRows / totalRows | Row counts |
estimatedCompletion | ISO 8601 timestamp |
8. Validating Bulk Data Format
| Stage | Validation |
|---|---|
| Pre-flight | File size, MIME type, encoding |
| Header | Required columns present |
| Per-row | Schema, types, business rules |
| Cross-row | Uniqueness, references |
9. Handling Partial Import Failures
| Strategy | Result |
|---|---|
| Continue on error | Skip bad rows, report at end |
| Stop on first error | Reject entire import |
| Error report download | CSV of failed rows + reasons |
| Resume from offset | Replay after fix |
10. Implementing Data Streaming
| Direction | Tech |
|---|---|
| Server → Client | Chunked transfer encoding, SSE, WebSocket |
| Client → Server | Chunked upload, multipart streaming |
| Both | HTTP/2 bidirectional, gRPC streaming |