Implementing Retry Mechanisms

1. Understanding Retry Behavior

Aspect Description Detail
Auto Retry Producer retries retriable errors Built-in
Bounded By delivery.timeout.ms Hard cap
Idempotence Prevents dup on retry Safe

Example: Retry config

props.put("retries", Integer.MAX_VALUE);
props.put("delivery.timeout.ms", 120000);

2. Configuring Retry Count

Config Description Default
retries Max resend attempts Integer.MAX
Effective Cap delivery.timeout.ms governs Time-bound
0 No retries, fail fast At-most-once

Example: Limit retries

props.put("retries", 5);

3. Setting Retry Backoff

Config Description Default
retry.backoff.ms Wait between retries 100
retry.backoff.max.ms Cap for exponential backoff NEW 1000
Avoids Retry storms Backpressure

Example: Backoff config

props.put("retry.backoff.ms", 200);
props.put("retry.backoff.max.ms", 5000);

4. Configuring Delivery Timeout

Config Description Default
delivery.timeout.ms Total send deadline 120000
Includes Batching + retries + inflight Whole lifecycle
Expiry TimeoutException to callback Final failure

Example: Set delivery deadline

props.put("delivery.timeout.ms", 60000);

5. Understanding Retriable Exceptions

Exception Cause Retriable
NotEnoughReplicas ISR below min Yes
LeaderNotAvailable Election in progress Yes
NetworkException Transient network Yes

Example: Check retriable

if (ex instanceof RetriableException) scheduleRetry(record);
else sendToDlq(record, ex);

6. Handling Non-Retriable Errors

Exception Cause Action
RecordTooLarge Exceeds max size Reject/split
SerializationException Bad payload DLQ
AuthorizationException ACL denied Fix perms

Example: Route non-retriable to DLQ

producer.send(record, (md, ex) -> {
    if (ex != null && !(ex instanceof RetriableException))
        dlqProducer.send(toDlq(record, ex));
});

7. Implementing Exponential Backoff

Aspect Description Detail
Formula delay = base × 2^attempt Capped
Jitter Random spread Avoid thundering herd
Built-in Producer backoff is exponential Since 3.7

Example: Consumer-side backoff

long delay = Math.min(maxMs, baseMs * (1L << attempt));
delay += ThreadLocalRandom.current().nextLong(delay / 2); // jitter
Thread.sleep(delay);

8. Using Circuit Breaker Pattern

State Behavior Detail
Closed Normal flow Counts failures
Open Fail fast, skip downstream After threshold
Half-Open Trial requests Probe recovery

Example: Circuit breaker around sink

if (breaker.allowRequest()) {
    try { sink.write(r); breaker.success(); }
    catch (Exception e) { breaker.failure(); pause(r); }
}

9. Configuring Request Timeout

Config Description Default
request.timeout.ms Per-request broker wait 30000
Timeout Counts as retriable failure May retry
Bound ≤ delivery.timeout.ms Constraint

Example: Request timeout

props.put("request.timeout.ms", 25000);

10. Implementing Consumer Retry Logic

Pattern Description Detail
Retry Topic Re-publish to delayed topic Non-blocking
In-place Retry Pause partition, retry Blocking
DLQ After max attempts Final

Example: Retry-topic pattern

try { process(r); }
catch (Exception e) {
    int n = retryCount(r);
    if (n < MAX) producer.send(new ProducerRecord<>("orders.retry", r.key(), r.value()));
    else producer.send(new ProducerRecord<>("orders.dlq", r.key(), r.value()));
}

Related Cheat Sheets

Kafka

Implementing Transactions

Kafka

Using Kafka CLI Tools

Kafka

Working with Time-Based Processing

Kafka

Using Admin API