Optimizing Producer Performance

1. Tuning Batch Size

Config Description Default
batch.size Max batch bytes/partition 16384
Larger Higher throughput More memory
Full Batch Sent immediately Regardless of linger

Example: Larger batch

props.put("batch.size", 65536);

2. Configuring Linger Time

Config Description Default
linger.ms Wait to fill batch 0
Tradeoff Latency vs throughput Higher = fewer requests
Typical 5-20ms throughput Batching

Example: Linger for batching

props.put("linger.ms", 10);

3. Optimizing Compression

Codec Ratio CPU
lz4 Good Low
zstd Best Medium
snappy Moderate Very low

Example: Enable compression

props.put("compression.type", "zstd");

4. Setting Buffer Memory

Config Description Default
buffer.memory Total send buffer 33554432
max.block.ms Block when full 60000
Backpressure send() blocks if full Flow control

Example: Buffer memory

props.put("buffer.memory", 67108864);

5. Configuring Max In-Flight Requests

Config Description Default
max.in.flight.
requests.per.connection		 Unacked requests 5
Idempotence ≤5 keeps ordering Safe
Higher More throughput Reorder risk

Example: In-flight requests

props.put("max.in.flight.requests.per.connection", 5);

6. Using Idempotent Producer

Aspect Description Detail
enable.idempotence Exactly-once per partition Default true
Requires acks=all, retries>0 Auto-set
Cost Negligible Recommended

Example: Idempotence

props.put("enable.idempotence", true);

7. Optimizing Serialization

Aspect Description Detail
Binary Avro/Protobuf compact vs JSON
Reuse Cache serializers No realloc
Schema ID Small wire footprint Registry

Example: Compact serializer

props.put("value.serializer",
  "io.confluent.kafka.serializers.KafkaAvroSerializer");

8. Tuning Request Timeout

Config Description Default
request.timeout.ms Per-request wait 30000
Too Low False timeouts Spurious retry
Bound ≤ delivery.timeout.ms Constraint

Example: Request timeout

props.put("request.timeout.ms", 30000);

9. Configuring Retries

Config Description Detail
retries Resend attempts Bounded by timeout
acks=all Durability Pair with retries
Idempotence No dup on retry Enable

Example: Durable producer

props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);

10. Monitoring Producer Metrics

Metric Description Watch
record-send-rate Throughput Trend
batch-size-avg Batching efficiency Tune linger
record-error-rate Failures Alert

Example: Producer JMX

kafka.producer:type=producer-metrics,client-id=app
  -> record-send-rate, batch-size-avg

Related Cheat Sheets

Kafka

Monitoring Consumer Metrics

Kafka

Configuring Quotas

Kafka

Monitoring Producer Metrics

Kafka

Understanding Message Timestamps