Designing Data Storage Architecture
1. Choosing Object Storage
| Property | Object Storage |
|---|---|
| Examples | S3, GCS, Azure Blob, MinIO |
| API | HTTP REST (PUT/GET/DELETE) |
| Durability | 11 nines (S3 standard) |
| Throughput | Massively parallel |
| Limits | No partial updates; eventual list |
| Best for | Backups, media, data lake, large blobs |
2. Designing Block Storage Architecture
| Property | Detail |
|---|---|
| Examples | EBS, GCE PD, Azure Disk, iSCSI |
| Access | Mount as block device, format FS |
| IOPS | Provisioned (gp3, io2) |
| Best for | Databases, OS volumes |
| Snapshots | Incremental, copy-on-write |
3. Designing File Storage Systems
| Type | Examples | Use Case |
|---|---|---|
| NFS | EFS, GCP Filestore | Shared POSIX |
| SMB | FSx for Windows | Windows shares |
| Parallel FS | Lustre, GPFS | HPC |
| Local FS | ext4, xfs, zfs | Single-node |
4. Designing Distributed File Systems
| System | Model |
|---|---|
| HDFS | Master (NameNode) + DataNodes; 3× replication |
| Ceph | CRUSH map; object/block/file |
| GlusterFS | No metadata server, elastic hashing |
| JuiceFS / Alluxio | POSIX over object storage |
5. Designing Data Lake Architecture
| Layer | Component |
|---|---|
| Storage | S3 / GCS / ADLS |
| Format | Parquet, ORC, Avro |
| Catalog | Glue, Hive Metastore, Unity Catalog |
| Compute | Spark, Trino, Presto, Athena |
| Zones | Raw / Cleansed / Curated (medallion) |
6. Designing Data Lakehouse Architecture
| Capability | Tech |
|---|---|
| ACID on lake | Delta Lake, Iceberg, Hudi |
| Time travel | Snapshot versions, AS OF queries |
| Schema evolution | Add/rename columns safely |
| Streaming ingest | Structured Streaming, Flink |
| Engine | Spark, Trino, DuckDB |
7. Designing Cold vs Hot Storage Strategy
| Tier | Cost | Access Latency | Use |
|---|---|---|---|
| Hot (SSD/standard) | $$$ | ms | Active data |
| Warm (IA, Nearline) | $$ | ms | Infrequent |
| Cold (Glacier, Coldline) | $ | min–hours | Backups |
| Archive (Deep) | ¢ | hours–12h | Compliance |
8. Designing Data Archival Strategy
| Element | Recommendation |
|---|---|
| Retention policy | Per data class; compliance-driven |
| Lifecycle rules | S3 lifecycle: hot → IA → Glacier |
| Format | Compressed Parquet/Avro |
| Encryption | SSE-KMS at rest |
| Restore SLA | Document expected RTO |
9. Designing Blob Storage Patterns
| Pattern | Description |
|---|---|
| Pre-signed URLs | Direct upload/download from client |
| Multipart upload | Files >100MB; parallel parts |
| Resumable upload | Network-tolerant uploads |
| Content-addressed | SHA-256 as key for dedup |
| Server-side processing | S3 Object Lambda, GCS triggers |
10. Designing Storage Tiering
| Strategy | Trigger |
|---|---|
| Time-based | Move after N days unaccessed |
| Access pattern | S3 Intelligent-Tiering moves automatically |
| Manual | Lifecycle rules per prefix |
| Tag-based | Move based on object tags |
11. Designing Storage Replication Strategy
| Type | Description |
|---|---|
| Same-region | 3+ AZs, durability |
| Cross-region | S3 CRR; DR + low-latency reads |
| Cross-account | Compliance, isolation |
| Replication metrics | Replication lag monitor |
12. Designing Object Versioning Strategy
| Feature | Notes |
|---|---|
| Versioning enabled | Each PUT creates new version |
| Object Lock | WORM compliance (governance/compliance mode) |
| Lifecycle | Expire noncurrent versions after N days |
| MFA delete | Protects against accidental delete |