Monitoring Producer Metrics

1. Tracking Record Send Rate

Metric Description Detail
record-send-rate Records/sec sent Throughput
record-send-total Cumulative count Lifetime
Per-Topic topic tag available Breakdown

Example: Send rate

kafka.producer:type=producer-metrics,client-id=app -> record-send-rate

2. Monitoring Batch Size

Metric Description Detail
batch-size-avg Average batch bytes Batching
Low vs Config Increase linger.ms Better batching
records-per-request Records per batch Efficiency

Example: Batch size avg

kafka.producer:type=producer-metrics,client-id=app -> batch-size-avg

3. Checking Compression Ratio

Metric Description Detail
compression-rate-avg Compressed/uncompressed Lower better
Effectiveness Codec choice impact zstd vs lz4
Tradeoff Ratio vs CPU Balance

Example: Compression ratio

kafka.producer:type=producer-metrics,client-id=app -> compression-rate-avg

4. Monitoring Record Queue Time

Metric Description Detail
record-queue-time-avg Time in buffer Before send
High Buffer pressure Throttling
Indicates Producer can't keep up Scale

Example: Queue time

kafka.producer:type=producer-metrics,client-id=app -> record-queue-time-avg

5. Tracking Request Latency

Metric Description Detail
request-latency-avg Broker round-trip Network+broker
request-latency-max Worst case Tail
Spikes Broker/network issue Correlate

Example: Request latency

kafka.producer:type=producer-metrics,client-id=app -> request-latency-avg

6. Monitoring Buffer Available Bytes

Metric Description Detail
buffer-available-bytes Free send buffer Headroom
Near Zero Buffer exhaustion Blocking risk
buffer-total-bytes Configured total Reference

Example: Buffer available

kafka.producer:type=producer-metrics,client-id=app -> buffer-available-bytes

7. Checking Waiting Threads

Metric Description Detail
waiting-threads Blocked on buffer Backpressure
Non-Zero send() blocking Slow down
Fix Increase buffer/scale Relieve

Example: Waiting threads

kafka.producer:type=producer-metrics,client-id=app -> waiting-threads

8. Tracking Record Error Rate

Metric Description Detail
record-error-rate Failed sends/sec Alert >0
record-error-total Cumulative errors Trend
Cause Auth, size, timeout Diagnose

Example: Error rate

kafka.producer:type=producer-metrics,client-id=app -> record-error-rate

9. Monitoring Record Retry Rate

Metric Description Detail
record-retry-rate Retries/sec Transient issues
High Broker instability Investigate
Correlate With ISR/network Root cause

Example: Retry rate

kafka.producer:type=producer-metrics,client-id=app -> record-retry-rate

10. Checking Metadata Age

Metric Description Detail
metadata-age Seconds since refresh Staleness
High Stale topology Refresh issue
Bound metadata.max.age.ms Config cap

Example: Metadata age

kafka.producer:type=producer-metrics,client-id=app -> metadata-age

Related Cheat Sheets

Kafka

Managing Schema Evolution

Kafka

Configuring Kafka Streams

Kafka

Using KRaft Consensus Protocol

Kafka

Configuring Kafka Connect