Building Fault Tolerance and Resilience
1. Implementing Circuit Breaker Pattern
failures >= threshold
┌─────────┐ ─────────────────────▶ ┌──────┐
│ CLOSED │ │ OPEN │
│ (pass) │ ◀───────────────────── │ (fail│
└─────────┘ success in HALF-OPEN │ fast)│
▲ └──────┘
│ success │
┌────┴─────┐ │ after timeout
│HALF-OPEN │ ◀──────────────────────────┘
│(probe) │
└──────────┘
| State | Behavior |
|---|
| CLOSED | Requests pass; count failures |
| OPEN | Reject immediately |
| HALF-OPEN | Allow test requests; success → CLOSED |
| Libraries | Resilience4j, Hystrix (deprecated), Polly, sentinel |
Example: Resilience4j circuit breaker
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
.failureRateThreshold(50)
.slowCallDurationThreshold(Duration.ofSeconds(2))
.waitDurationInOpenState(Duration.ofSeconds(30))
.slidingWindowSize(20)
.build();
CircuitBreaker cb = CircuitBreaker.of("paymentService", config);
Supplier<Payment> safe = CircuitBreaker.decorateSupplier(cb, () -> client.charge());
Payment result = Try.ofSupplier(safe).recover(t -> fallback()).get();
2. Implementing Retry Strategies
| Strategy | Formula | Use |
|---|
| Fixed delay | delay = c | Predictable |
| Linear | delay = c × n | Mild backoff |
| Exponential | delay = base × 2^n | Standard |
| Exponential + jitter | random(0, base × 2^n) | Avoids thundering herd |
| Decorrelated jitter | random(base, prev × 3) | AWS recommendation |
Warning: Only retry idempotent or retry-safe operations; do not retry 4xx errors.
3. Implementing Timeout Strategies
| Type | Detail |
|---|
| Connect timeout | Time to establish TCP/TLS |
| Request timeout | Bound total call |
| Read/idle timeout | Inter-byte gap |
| Deadline propagation | gRPC deadline header — passed downstream |
| Budget-based | Remaining budget split among hops |
4. Implementing Bulkhead Pattern
| Variant | Detail |
|---|
| Thread pool isolation | Per-dependency pool prevents pool exhaustion spreading |
| Semaphore | Cap concurrent calls per dependency |
| Process / container isolation | Separate deployments |
| Network bulkhead | Per-tenant queues / rate limits |
5. Implementing Failover Mechanisms
| Type | Trigger |
|---|
| Active-passive | Standby promoted on health loss |
| Active-active | Traffic redistributed |
| DNS failover | TTL-bounded; Route 53 health checks |
| Anycast | BGP withdraws unhealthy region |
6. Implementing Graceful Degradation
| Tactic | Detail |
|---|
| Fallback values | Stale cache, default config |
| Feature toggle off | Disable non-essential features |
| Read-only mode | Reject writes; serve reads |
| Reduced fidelity | Lower-res images, skip recommendations |
7. Implementing Chaos Engineering Practices
| Tool | Capability |
|---|
| Chaos Monkey | Random instance termination |
| Chaos Mesh | K8s-native; pod, network, IO faults |
| Litmus | K8s chaos workflows |
| Gremlin | Commercial, broad fault injection |
| AWS FIS | Managed fault injection |
8. Understanding Failure Detection Mechanisms
| Mechanism | Detail |
|---|
| Heartbeat / push | Periodic alive signal |
| Probe / pull | Health check ping |
| Phi accrual | Adaptive suspicion level |
| SWIM | Indirect probe via peers |
| Outlier detection | Statistical (Envoy) |
9. Implementing Self-Healing Systems
| Component | Action |
|---|
| Liveness probe | Restart container on failure |
| Auto-scaling | Replace unhealthy instances (ASG) |
| Operator pattern | Reconcile desired vs actual state |
| Rolling restart | Periodic restart to mitigate leaks |
10. Handling Cascading Failures
| Cause | Mitigation |
|---|
| Retry storm | Limit retries; jittered backoff |
| Synchronous coupling | Async messaging; circuit breaker |
| Resource exhaustion | Bulkhead; admission control |
| Saturated dependency | Load shedding; degrade gracefully |
| DNS / config flap | TTL caching; gradual rollout |
11. Implementing Rate Limiting and Throttling
| Algorithm | Behavior |
|---|
| Token bucket | Bursts allowed up to bucket size; refills at rate |
| Leaky bucket | Smooths to constant outflow |
| Fixed window | N reqs per minute; boundary spikes |
| Sliding window log | Precise; memory per request |
| Sliding window counter | Approximation; common in practice |
| Distributed (Redis) | INCR + EXPIRE; Lua scripts |