Debugging and Troubleshooting

1. Enabling Debug Logging

Aspect Description Detail
log4j2 Per-package level Granular
Client Logger org.apache.kafka DEBUG
Dynamic kafka-configs log level No restart

Example: Dynamic log level

kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type broker-loggers --entity-name 1 \
  --add-config kafka.request.logger=DEBUG

2. Using Broker Logs

Log Description Detail
server.log Main broker log Errors/info
kafkaServer-gc GC pauses Latency cause
grep ERROR Find failures Triage

Example: Tail server log

tail -f /var/lib/kafka/logs/server.log | grep -i error

3. Checking Controller Logs

Aspect Description Detail
controller.log Leader/metadata events KRaft
Election Quorum changes Failover
Metadata Lag Apply delays Health

Example: Controller log grep

grep -i "controller\|election" \
  /var/lib/kafka/logs/controller.log

4. Analyzing State Change Logs

Aspect Description Detail
state-change.log Partition transitions Leader/ISR
Replica States Online/offline Diagnose URP
Timeline Correlate incidents Forensics

Example: State change for topic

grep "orders-0" /var/lib/kafka/logs/state-change.log

5. Using Kafka Dump Log Tool

Aspect Description Detail
Inspect Segments Decode log files Low-level
--print-data-log Show record contents Payload
--deep-iteration Verify each record Corruption

Example: Dump segment

kafka-dump-log.sh --files \
  /var/lib/kafka/logs/orders-0/00000000000000000000.log \
  --print-data-log

6. Checking Topic Metadata

Aspect Description Detail
--describe Leader/ISR/replicas Health
Leader -1 No leader (offline) Problem
ISR Shrink Replica lagging Investigate

Example: Describe topic

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

7. Analyzing Consumer Lag

Aspect Description Detail
LAG Column end - committed Backlog
Growing Consumer too slow Scale
Stuck Poison pill/hang Thread dump

Example: Describe group lag

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group orders-svc

8. Using JMX Console

Aspect Description Detail
JConsole Live MBean browse Interactive
jmxterm CLI query Scriptable
JMX_PORT Enable remote Env var

Example: Enable JMX

export JMX_PORT=9999
# Then connect: jconsole localhost:9999

9. Tracing Request Path

Aspect Description Detail
Request Logger Per-request timing DEBUG
RequestQueueTime Queue vs process Breakdown
Client Trace correlation id Cross-service

Example: Request log timing fields

totalTime, requestQueueTime, localTime,
remoteTime, responseQueueTime, sendTime (ms)

10. Debugging Rebalance Issues

Symptom Description Detail
Frequent Rebalance Slow processing max.poll exceeded
Fix Raise max.poll.interval Or shrink batch
Static Membership group.instance.id Avoid churn

Example: Stabilize rebalances

max.poll.interval.ms=600000
max.poll.records=100
group.instance.id=consumer-1

Related Cheat Sheets

Kafka

Using Schema Registry

Kafka

Understanding Message Ordering

Kafka

Working with Time-Based Processing

Kafka

Configuring Consumer Settings