Configuring Producer Settings

1. Setting Batch Size

Config Description Default
batch.size Max bytes per partition batch 16384
Larger Better throughput, more memory 32–64KB common
Full Batch Sent immediately when full Ignores linger

Example: Larger batches

props.put("batch.size", 65536);   // 64KB
props.put("linger.ms", 10);

2. Configuring Linger Time

Config Description Default
linger.ms Wait time to fill a batch 0
Tradeoff Higher = better batching, more latency 5–100ms
Throughput Pairs with batch.size Combined tuning

Example: Batch for throughput

props.put("linger.ms", 20); // wait up to 20ms for fuller batches

3. Setting Buffer Memory

Config Description Default
buffer.memory Total bytes for unsent records 33554432 (32MB)
max.block.ms Block time when buffer full 60000
Backpressure send() blocks then throws TimeoutException

Example: Increase buffer

props.put("buffer.memory", 67108864); // 64MB
props.put("max.block.ms", 30000);

4. Configuring Compression

Type Ratio CPU
none 1x None
lz4 Good Low
zstd Best Medium
gzip High High

Example: Enable zstd

props.put("compression.type", "zstd");

5. Setting Max Request Size

Config Description Default
max.request.size Largest record/request bytes 1048576 (1MB)
Broker Limit Must align with message.max.bytes Both must allow
Oversize Throws RecordTooLargeException Rejected

Example: Allow larger messages

props.put("max.request.size", 5242880); // 5MB

6. Setting Request Timeout

Config Description Default
request.timeout.ms Wait for broker response 30000
Retry Trigger Timeout counts as failure May retry
Relation ≤ delivery.timeout.ms Bounded

Example: Set request timeout

props.put("request.timeout.ms", 20000);

7. Configuring Delivery Timeout

Config Description Default
delivery.timeout.ms Total time from send to success/fail 120000
Bound ≥ linger.ms + request.timeout.ms Upper limit
Includes Batching, retries, in-flight Whole lifecycle

Example: Cap total delivery time

props.put("delivery.timeout.ms", 60000);

8. Configuring Retries

Config Description Default
retries Resend attempts on failure Integer.MAX
retry.backoff.ms Wait between retries 100
Bound Limited by delivery.timeout.ms Effective cap

Example: Retry config

props.put("retries", Integer.MAX_VALUE);
props.put("retry.backoff.ms", 200);

9. Setting Idempotence

Config Description Default
enable.idempotence Exactly-once per partition on retry true NEW
Requires acks=all, retries>0, inflight≤5 Auto-set
PID Producer ID + sequence dedup Broker-side

Example: Idempotent producer

props.put("enable.idempotence", "true"); // prevents duplicates on retry

10. Configuring Max In-Flight Requests

Config Description Default
max.in.flight.
requests.per.connection		 Unacked requests per connection 5
Ordering ≤5 keeps order with idempotence Safe
Without Idempotence Set to 1 for strict order Slower

Example: Set in-flight limit

props.put("max.in.flight.requests.per.connection", 5);

Related Cheat Sheets

Kafka

Installing and Configuring Kafka

Kafka

Understanding Message Timestamps

Kafka

Monitoring Consumer Metrics

Kafka

Testing Kafka Applications