Handling Errors and Exceptions
1. Understanding Producer Exceptions
| Category | Description | Detail |
|---|---|---|
| Retriable | Transient, auto-retry | RetriableException |
| Non-Retriable | Fatal, no retry | Handle/DLQ |
| Delivery | Via callback/Future | Async |
2. Handling Timeout Exceptions
| Exception | Cause | Fix |
|---|---|---|
| TimeoutException | delivery.timeout exceeded | Increase/retry |
| Metadata Timeout | Broker unreachable | Check network |
| Buffer Full | max.block.ms hit | Slow down |
Example: Catch timeout
if (ex instanceof TimeoutException) {
metrics.timeout(); retryLater(record);
}
3. Dealing with Serialization Errors
| Aspect | Description | Detail |
|---|---|---|
| SerializationException | Bad payload/schema | Non-retriable |
| Producer | Fails before send | Validate first |
| Consumer | Poison pill | Skip/DLQ |
Example: Guard serialization
try { producer.send(record); }
catch (SerializationException e) { dlq.send(record); }
4. Handling Authorization Failures
| Exception | Cause | Fix |
|---|---|---|
| TopicAuthorizationException | No topic ACL | Grant access |
| GroupAuthorizationException | No group ACL | Add Read |
| Non-Retriable | Won't fix itself | Alert ops |
Example: Handle auth failure
catch (TopicAuthorizationException e) {
log.error("Missing ACL for {}", e.unauthorizedTopics());
}
5. Managing Retriable Exceptions
| Aspect | Description | Detail |
|---|---|---|
| Auto-Retry | Producer retries | Within timeout |
| Idempotence | Avoid dup on retry | enable |
| Backoff | Spacing between tries | retry.backoff.ms |
Example: Safe retries
props.put("enable.idempotence", true);
props.put("retries", Integer.MAX_VALUE);
6. Handling Non-Retriable Errors
| Aspect | Description | Detail |
|---|---|---|
| Detect | Not RetriableException | instanceof |
| Action | DLQ or alert | No retry |
| Examples | RecordTooLarge, auth | Fatal |
Example: Branch on retriability
if (ex instanceof RetriableException) retry(r);
else { dlq.send(r); alert(ex); }
7. Understanding Consumer Exceptions
| Exception | Cause | Fix |
|---|---|---|
| CommitFailedException | Rebalance during commit | Retry/handle |
| WakeupException | Intentional shutdown | Expected |
| RecordDeserialization | Poison record | Skip |
Example: Wakeup shutdown
try { while (running) consumer.poll(d); }
catch (WakeupException e) { /* shutdown */ }
finally { consumer.close(); }
8. Handling Offset Out of Range
| Config | Description | Detail |
|---|---|---|
| auto.offset.reset | earliest/latest/none | Recovery |
| Cause | Offset deleted by retention | Gap |
| none | Throws exception | Manual handle |
9. Dealing with Rebalance Exceptions
| Aspect | Description | Detail |
|---|---|---|
| CommitFailed | Partitions revoked | Commit in listener |
| Listener | onPartitionsRevoked | Final commit |
| Cooperative | Reduce disruption | Incremental |
Example: Commit before revoke
new ConsumerRebalanceListener() {
public void onPartitionsRevoked(Collection<TopicPartition> p){
consumer.commitSync();
}
};
10. Configuring Error Handlers
| Aspect | Description | Detail |
|---|---|---|
| UncaughtException | Thread crash policy | Streams |
| Global Handler | Catch-all logging | Centralize |
| Framework | Spring error handlers | DefaultErrorHandler |