Working with Distributed File Systems
1. Understanding DFS Architecture
| Component | Role | Examples |
|---|---|---|
| NameNode / Master | Metadata, namespace | HDFS NameNode, GFS Master |
| DataNode / Chunkserver | Stores data blocks | HDFS DataNode |
| Client | Reads/writes via library | HDFS client, S3 SDK |
| Object stores | Flat namespace, REST API | S3, GCS, Azure Blob |
| POSIX-like | Mountable filesystem | CephFS, GlusterFS, JuiceFS |
2. Implementing File Replication
| Method | Detail |
|---|---|
| Whole-file copy | N replicas of full file |
| Block-level (HDFS) | 3× by default; rack-aware placement |
| Erasure coding | k+m shards; tolerates m losses; ~1.5× overhead vs 3× |
| Cross-region | S3 CRR, geo-redundant storage |
3. Understanding Block Storage
| Aspect | Detail |
|---|---|
| Block size (HDFS) | 128MB or 256MB default |
| Object size (S3) | 5MB-5GB single PUT; multipart up to 5TB |
| Append-only | HDFS, GFS optimized for append |
| Random write | Limited; designed for batch |
4. Implementing Data Locality Optimization
| Principle | Detail |
|---|---|
| Move compute to data | Schedule task on node holding block |
| Locality levels | NODE_LOCAL > RACK_LOCAL > ANY |
| Used by | YARN, Spark, MapReduce |
| Disaggregated trend | Compute/storage separation (S3 + EMR) |
5. Handling File Consistency
| System | Model |
|---|---|
| HDFS | Single-writer, append-only; strong |
| S3 (since Dec 2020) | Strong read-after-write for all ops |
| CephFS | Strong via MDS |
| NFSv4 | Close-to-open consistency |
6. Implementing File Caching
| Layer | Detail |
|---|---|
| Client cache | Local disk / memory (Alluxio) |
| Page cache | OS-level read cache |
| CDN edge | Geo-distributed caching |
| S3 + CloudFront | Standard pattern |
7. Understanding Rack Awareness
| Placement Rule | Detail |
|---|---|
| HDFS replica 1 | Same node as writer (or random) |
| Replica 2 | Different rack |
| Replica 3 | Same rack as replica 2 (different node) |
| Goal | Survive single rack failure; minimize cross-rack traffic |
8. Implementing Data Integrity Checks
| Mechanism | Detail |
|---|---|
| Checksum (CRC32C, xxHash) | Per block, verified on read |
| End-to-end (S3) | MD5 / SHA-256 over PUT/GET |
| Background scrubbing | Periodic scan, repair from replica |
| Bit rot detection | ZFS/Btrfs checksums; ECC RAM |
9. Handling Node Failures
| Phase | Action |
|---|---|
| Detect | Heartbeat timeout (HDFS = 10.5 min default) |
| Re-replicate | NameNode schedules new copies for under-replicated blocks |
| Throttle | Limit MB/s to avoid network saturation |
| Decommission | Drain blocks before removal |
10. Understanding Read and Write Patterns
| Pattern | Optimization |
|---|---|
| Sequential read | Prefetching, large blocks |
| Random read | SSD-backed; row-oriented formats |
| Append-only | HDFS, Kafka, log-structured stores |
| Random write | Generally avoided in DFS |
| Write-once-read-many (WORM) | Compliance, archival |