Designing Resilience Patterns
1. Designing Timeout Patterns
| Layer | Detail |
|---|---|
| Connection | Time to establish socket (~1s) |
| Read / write | Per-call deadline |
| Overall request | End-to-end budget |
| Cascading deadlines | Pass remaining budget to downstream |
| Always set | Default infinite is dangerous |
2. Designing Retry with Exponential Backoff
| Aspect | Detail |
|---|---|
| Idempotency required | Only retry safe ops |
| Jitter | Full jitter recommended |
| Retry budget | Cap retries % to avoid amplification |
| Don't retry | 4xx (except 408/425/429) |
3. Designing Circuit Breaker Pattern
CLOSED ──fail>threshold──▶ OPEN
▲ │
│ wait reset timeout
success │
│ ▼
HALF-OPEN ◀────test─── (allow probe calls)
| State | Behavior |
|---|---|
| Closed | Calls pass; track failures |
| Open | Fail fast immediately |
| Half-open | Allow N probes; close on success |
| Libraries | Resilience4j, Polly, Hystrix (legacy) |
4. Designing Bulkhead Pattern
| Aspect | Detail |
|---|---|
| Idea | Isolate resources per dependency |
| Implementation | Separate thread pools / semaphores |
| Benefit | One slow dep can't exhaust all threads |
| K8s level | Separate deployments per critical service |
5. Designing Fallback Mechanisms
| Type | Example |
|---|---|
| Static default | Empty list / cached value |
| Stale cache | Serve last known-good |
| Alternate provider | Failover to secondary |
| Degraded UX | Hide non-essential block |
6. Designing Graceful Degradation
| Strategy | Detail |
|---|---|
| Feature flags | Disable expensive features |
| Read-only mode | Pause writes |
| Sampling | Reduce % of work |
| Quality reduction | Lower-res images, fewer recs |
7. Designing Health Check Architecture
| Endpoint | Purpose |
|---|---|
| /livez | Process alive (restart if fail) |
| /readyz | Ready to serve traffic |
| /startupz | Slow startup grace |
| Deep vs shallow | Avoid cascading failures via deep checks |
8. Designing Failure Detection
| Mechanism | Detail |
|---|---|
| Heartbeat | Periodic ping |
| Phi accrual detector | Adaptive (Akka, Cassandra) |
| Gossip | Members exchange state |
| SLO breach alerts | Symptom-based detection |
9. Designing Service Isolation
| Boundary | Detail |
|---|---|
| Process | Separate containers |
| Data | DB-per-service; no shared state |
| Network | NetworkPolicies, namespaces |
| Failure domain | Per AZ / cluster / region |
10. Designing Chaos Engineering Practices
| Aspect | Detail |
|---|---|
| Hypothesis-driven | Steady-state metric → inject failure |
| Tools | Chaos Mesh, LitmusChaos, Gremlin, AWS FIS |
| Game days | Scheduled team exercises |
| Start small | Staging → controlled prod |
| Automate rollback | Kill switch always available |
11. Designing Blast Radius Limitation
| Technique | Detail |
|---|---|
| Cell-based architecture | Independent cells; route per user/tenant |
| Shuffle sharding | Reduce cross-tenant impact |
| Canary + auto-rollback | Limit deploy blast radius |
| Quotas per tenant | Stop noisy neighbor |
12. Designing Self-Healing Systems
| Mechanism | Detail |
|---|---|
| K8s replicas | Auto-restart failed pods |
| Auto-scaling | HPA / VPA / Cluster Autoscaler |
| Self-healing operators | Reconcile to desired state |
| Auto-failover | DB replica promotion |
| Auto-remediation | Runbook automation on alerts |