Designing File Storage and Media Architecture
1. Designing File Storage Strategy
| Tier | Use |
|---|---|
| Hot (S3 Standard) | Frequent access |
| Infrequent (S3 IA) | Backups, older media |
| Glacier / Archive | Compliance, cold |
| Block storage | EBS/PD for VM disks |
| File systems | EFS/Filestore for shared POSIX |
2. Designing File Upload Architecture
| Pattern | Detail |
|---|---|
| Direct-to-S3 with presigned URL | Bypasses app servers |
| Multipart upload | Files > 100MB; parallel parts |
| Resumable (TUS) | For unreliable networks |
| Virus scan | S3 events → ClamAV / GuardDuty |
| MIME validation | Don't trust extension |
3. Designing File Access Control
| Mechanism | Detail |
|---|---|
| Presigned URLs | Short TTL, scoped permission |
| Signed cookies | For batch access in CDN |
| App-mediated | Stream through app layer (slower) |
| Bucket policy | Default deny; explicit allow |
| No public bucket | Unless intentional + reviewed |
4. Designing File Processing Pipeline
Upload → S3 → Event → Queue → Worker (transcode/resize/scan) → Output → DB metadata → CDN
| Stage | Detail |
|---|---|
| Trigger | S3 event / Pub/Sub |
| Worker pool | Auto-scaled, idempotent |
| Status | Track per asset (pending/done/failed) |
5. Designing CDN Integration for Files
| Practice | Detail |
|---|---|
| CDN in front of bucket | S3 + CloudFront / GCS + Cloud CDN |
| Origin Access Identity | Bucket private; CDN-only access |
| Long TTL + versioned URLs | Cache forever |
| Signed URLs at CDN edge | Per-user access |
6. Designing Media Transcoding
| Tool | Detail |
|---|---|
| FFmpeg | Universal encoder |
| AWS MediaConvert / Elemental | Managed, scalable |
| Mux / Coconut | SaaS |
| Output ladder | Multi-bitrate (240p–4K) |
| Format | HLS / DASH for adaptive streaming |
7. Designing Image Optimization
| Technique | Detail |
|---|---|
| On-the-fly resize | imgproxy, Thumbor, Cloudflare Images |
| Format negotiation | Serve AVIF/WebP via Accept |
| Responsive srcset | Per device |
| Lazy loading | loading="lazy" |
| Strip metadata | EXIF removal for privacy + size |
8. Designing File Metadata Management
| Field | Detail |
|---|---|
| id, owner, size, type | Core |
| checksum | SHA-256 for dedup / integrity |
| tags / labels | For search / lifecycle |
| storage_class | Track tier |
| created_at / accessed_at | For lifecycle |
9. Designing File Lifecycle Management
| Rule | Detail |
|---|---|
| Tier transitions | 30d → IA, 90d → Glacier |
| Expiration | Auto-delete after N days |
| Versioning | Keep N versions |
| Object lock | Compliance retention |
10. Designing File Backup and Archival
| Practice | Detail |
|---|---|
| Cross-region replication | S3 CRR / GCS dual-region |
| Versioning | Recover deleted |
| Object lock | WORM / ransomware proof |
| Deep archive | S3 Glacier Deep Archive |
| Restore SLA | Hours for archive tier |
11. Designing Video Streaming Architecture
| Element | Detail |
|---|---|
| Protocol | HLS (Apple), MPEG-DASH |
| Segments | 2–6s chunks |
| DRM | Widevine, FairPlay, PlayReady |
| Live | LL-HLS, WebRTC for ultra-low latency |
| CDN | Essential for scale |
12. Designing Large File Handling
| Technique | Detail |
|---|---|
| Multipart upload | 5MB+ parts; parallel |
| Resumable upload | TUS protocol |
| Streaming download | Range requests |
| Checksum per part | Detect corruption |
| Avoid app server proxy | Direct to/from object store |