Implementing Circuit Breaker Patterns
1. Circuit Breaker Pattern
| Aspect | Detail |
|---|---|
| Purpose | Prevent cascading failure by failing fast when downstream is unhealthy |
| Wrap | Each remote call |
| Tools | Resilience4j, Hystrix (legacy), Polly (.NET), Istio outlier detection |
2. Circuit Breaker States
| State | Behavior | Transition |
|---|---|---|
| CLOSED | Calls pass; tracks failure rate | Failures > threshold → OPEN |
| OPEN | Calls fail fast (no remote attempt) | After cool-down → HALF-OPEN |
| HALF-OPEN | Permits N probe calls | Success → CLOSED; Failure → OPEN |
State Machine
CLOSED ──failures>threshold──> OPEN ──cool-down──> HALF-OPEN
^ │
└──────success in HALF-OPEN─────────────────────────┘
(failure → OPEN)
3. Circuit Breaker Fallback Pattern
| Fallback Strategy | Example |
|---|---|
| Cached Last-Known | Return last successful response |
| Default Value | Empty list, zero, "not available" |
| Alternative Service | Failover to secondary provider |
| Degraded Mode | Skip non-critical feature |
| Fail Fast Error | HTTP 503 with Retry-After |
4. Circuit Breaker Threshold Configuration
| Param | Typical | Effect |
|---|---|---|
| failureRateThreshold | 50% | % of failures to trip |
| slowCallRateThreshold | 100% | % slow calls counted as failure |
| slowCallDurationThreshold | 2s | Above this = "slow" |
| slidingWindowSize | 100 calls | Sample size for stats |
| minimumNumberOfCalls | 20 | Min calls before stats considered |
| waitDurationInOpenState | 30s | Cool-down before HALF-OPEN |
| permittedCallsInHalfOpen | 5 | Probe count |
Example: Resilience4j Config
resilience4j.circuitbreaker:
instances:
paymentService:
failureRateThreshold: 50
slowCallDurationThreshold: 2s
slidingWindowSize: 100
minimumNumberOfCalls: 20
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 55. Circuit Breaker with Timeout
| Pattern | Detail |
|---|---|
| Combine | Timeout + circuit breaker (slow calls trip too) |
| Order | Timeout → fail call → counted by breaker |
| Tuning | Timeout shorter than upstream timeout in chain |
6. Hierarchical Circuit Breaker Pattern
| Level | Scope |
|---|---|
| Per Endpoint | Trip only failing endpoint |
| Per Service | Aggregate across endpoints |
| Per Cluster/Region | Failover at regional level |
7. Circuit Breaker Metrics Pattern
| Metric | Use |
|---|---|
| state (CLOSED/OPEN/HALF_OPEN) | Dashboards / alerts on OPEN |
| failureRate | Trend analysis |
| slowCallRate | Latency degradation |
| trip count | Outage frequency |
| callsNotPermitted | Requests rejected while open |
8. Circuit Breaker Recovery Pattern
| Step | Action |
|---|---|
| Cool Down | OPEN for fixed duration |
| Probe | HALF-OPEN allows limited test calls |
| Gradual Re-Open | Linear/exponential ramp on success |
| Manual Reset | Operator override via mgmt endpoint |
9. Per-Host Circuit Breaker Pattern
| Aspect | Detail |
|---|---|
| Definition | Independent breaker per upstream instance/IP |
| Benefit | Isolates one bad replica without tripping whole service |
| Implementation | Envoy outlier detection, Istio |
10. Adaptive Circuit Breaker Pattern
| Aspect | Detail |
|---|---|
| Adapts | Thresholds change based on observed traffic patterns |
| Examples | SREcon adaptive concurrency, Netflix concurrency-limits |
| Algorithm | TCP Vegas-like, Little's Law |
| Benefit | No static tuning; reacts to real load |