Implementing Error Recovery Patterns
1. Using Circuit Breaker Pattern
| State | Behavior |
|---|---|
| Closed | Calls pass through; track failures |
| Open | Fail fast with codes.Unavailable |
| Half-Open | Probe with limited traffic |
Example: sony/gobreaker
cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
Name: "user-svc", Timeout: 30*time.Second,
ReadyToTrip: func(c gobreaker.Counts) bool { return c.ConsecutiveFailures > 5 },
})
_, err := cb.Execute(func() (any, error) { return client.GetUser(ctx, req) })2. Implementing Retry Logic
| Aspect | Detail |
|---|---|
| Built-in | gRPC service-config retry policy (Section 32) |
| Custom | Exponential backoff + jitter |
| Only idempotent | GET/list/idempotent updates |
3. Implementing Fallback Strategies
| Fallback | Use |
|---|---|
| Cached value | Recent successful response |
| Default value | Safe minimal result |
| Degraded service | Reduced functionality |
4. Using Bulkhead Pattern
| Mechanism | Detail |
|---|---|
| Per-dependency goroutine pool | Limit concurrency to a downstream |
| Per-tenant quota | Prevent noisy neighbor |
| Semaphore | chan struct{} sized to N |
5. Implementing Timeout Patterns
| Pattern | Detail |
|---|---|
| Budget propagation | Parent deadline minus margin per hop |
| Hedged requests | Send second request after p99 |
| Adaptive | Increase timeout under sustained latency |
6. Handling Partial Failures
| Pattern | Detail |
|---|---|
| Best-effort batch | Return per-item status |
| Compensating action | Roll back successful sub-ops on failure |
| Eventual consistency | Outbox + async retry |
7. Using Dead Letter Queues
| Step | Detail |
|---|---|
| Retry N times | With backoff |
| On final failure | Enqueue to DLQ topic |
| Manual replay | Operator tool or scheduled job |
8. Implementing Compensating Transactions
| Saga Step | Compensation |
|---|---|
| Reserve inventory | Release inventory |
| Charge payment | Refund payment |
| Ship order | Cancel shipment |
9. Using Health Checks for Recovery
| Tier | Action |
|---|---|
| Liveness fail | Restart pod |
| Readiness fail | Remove from LB |
| gRPC Health watch | Client switches subchannel |
10. Implementing Graceful Degradation
| Technique | Detail |
|---|---|
| Feature flags | Disable non-critical features under load |
| Reduced payload | Return summary instead of full data |
| Static fallback | Serve cached/precomputed content |