Implementing Retry Logic
1. Configuring Retry Policy
Example: Built-in retry via service config
{
"methodConfig": [{
"name": [{ "service": "user.v1.UserService" }],
"retryPolicy": {
"maxAttempts": 4,
"initialBackoff": "0.1s",
"maxBackoff": "1s",
"backoffMultiplier": 2.0,
"retryableStatusCodes": ["UNAVAILABLE", "DEADLINE_EXCEEDED"]
}
}]
}| Field | Detail |
|---|---|
maxAttempts | Total attempts incl. first (max 5 by default) |
backoffMultiplier | Exponential factor |
retryableStatusCodes | Codes that trigger retry |
2. Setting Max Attempts
| Recommendation | Detail |
|---|---|
| 3–4 | Typical |
| Avoid >5 | Amplifies load on degraded backend |
| Bounded by deadline | Total time = parent ctx deadline |
3. Configuring Initial Backoff
| Setting | Detail |
|---|---|
| 0.1s | Typical for in-DC |
| Jitter | gRPC automatically applies random factor |
4. Configuring Max Backoff
| Setting | Detail |
|---|---|
| 1–10s | Cap to prevent unbounded growth |
| Per-attempt | Combined with multiplier and jitter |
5. Setting Retryable Status Codes
| Code | Retryable? |
|---|---|
| UNAVAILABLE | Yes — typical |
| DEADLINE_EXCEEDED | Yes if idempotent |
| RESOURCE_EXHAUSTED | Sometimes; honor RetryInfo |
| INTERNAL / UNKNOWN | Avoid unless explicitly safe |
| INVALID_ARGUMENT, NOT_FOUND | Never |
6. Using Hedging Policy
Example: Hedging
"hedgingPolicy": {
"maxAttempts": 3,
"hedgingDelay": "0.05s",
"nonFatalStatusCodes": ["UNAVAILABLE"]
}| Behavior | Detail |
|---|---|
| Parallel sends | After delay, send extra attempts in parallel |
| First wins | Cancel others on success |
| Requires idempotency | Server may receive multiple copies |
7. Implementing Custom Retry Logic
Example: Custom retry interceptor
func Retry(max int) grpc.UnaryClientInterceptor {
return func(ctx context.Context, m string, req, reply any,
cc *grpc.ClientConn, inv grpc.UnaryInvoker, opts ...grpc.CallOption) error {
var err error
backoff := 100 * time.Millisecond
for i := 0; i < max; i++ {
err = inv(ctx, m, req, reply, cc, opts...)
if err == nil { return nil }
if status.Code(err) != codes.Unavailable { return err }
select {
case <-time.After(backoff + time.Duration(rand.Int63n(int64(backoff)))):
case <-ctx.Done(): return ctx.Err()
}
backoff *= 2
}
return err
}
}8. Setting Per-Attempt Timeout
| Need | Approach |
|---|---|
| Bound each try | Wrap call with child ctx (timeout / maxAttempts) |
| Total budget | Parent ctx deadline applies to whole policy |
9. Using Exponential Backoff
| Schedule | Detail |
|---|---|
| Initial 100 ms | ×2 each attempt → 100, 200, 400, 800 ms |
| Cap at maxBackoff | Plateau prevents runaway |
| Add jitter | Prevents thundering herd |
10. Handling Non-Idempotent Methods
| Strategy | Detail |
|---|---|
| No retry | Default safest |
| Idempotency key | Client-generated UUID in request; server dedupes |
| Annotate | option idempotency_level = IDEMPOTENT; |