Optimizing Consumer Performance

1. Tuning Fetch Size

Config Description Default
fetch.min.bytes Min data before return 1
fetch.max.bytes Max per fetch 52428800
max.partition.
fetch.bytes		 Per-partition cap 1048576

Example: Larger fetch

props.put("fetch.min.bytes", 65536);

2. Configuring Fetch Wait Time

Config Description Default
fetch.max.wait.ms Wait for min.bytes 500
Tradeoff Latency vs efficiency Batch fill
Pair With fetch.min.bytes Together

Example: Fetch wait

props.put("fetch.max.wait.ms", 100);

3. Setting Max Poll Records

Config Description Default
max.poll.records Records per poll() 500
Lower Faster processing loop Avoid timeout
Higher More throughput Longer processing

Example: Poll records

props.put("max.poll.records", 200);

4. Optimizing Partition Assignment

Strategy Description Detail
CooperativeSticky Minimal movement Recommended
RoundRobin Even spread Multi-topic
Range Per-topic contiguous Default legacy

Example: Assignment strategy

props.put("partition.assignment.strategy",
  "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

5. Using Parallel Processing

Approach Description Detail
More Consumers Up to partition count Scale out
Worker Pool Offload processing Decouple poll
Order Risk Track offsets carefully Per partition

Example: Worker pool

for (var r : consumer.poll(d))
  executor.submit(() -> process(r));
// commit only after workers finish for that batch

6. Configuring Prefetching

Aspect Description Detail
Built-in Fetcher prefetches Background
receive.buffer.bytes TCP buffer Throughput
Pipeline Fetch while processing Overlap

Example: Receive buffer

props.put("receive.buffer.bytes", 1048576);

7. Optimizing Deserialization

Aspect Description Detail
Lazy Deserialize on demand Skip unused
Cache Schemas Avoid registry calls Local cache
Binary Faster than JSON Avro/Protobuf

Example: Cache size for registry

props.put("schema.registry.url", "http://localhost:8081");
props.put("max.schemas.per.subject", 1000);

8. Tuning Commit Frequency

Aspect Description Detail
auto.commit.
interval.ms		 Auto-commit period 5000
Less Frequent Lower overhead More reprocess
Manual Control on success Precise

Example: Commit interval

props.put("auto.commit.interval.ms", 10000);

9. Using Asynchronous Commits

Method Description Detail
commitAsync Non-blocking Higher throughput
commitSync Blocking + retry On shutdown
Hybrid Async in loop, sync at end Best practice

Example: Async + sync hybrid

try {
  while (running) { consume(); consumer.commitAsync(); }
} finally { consumer.commitSync(); consumer.close(); }

10. Monitoring Consumer Metrics

Metric Description Watch
records-lag-max Max backlog Scale signal
fetch-rate Fetches/sec Health
records-consumed-rate Throughput Trend

Example: Consumer lag JMX

kafka.consumer:type=consumer-fetch-manager-metrics,client-id=app
  -> records-lag-max

Related Cheat Sheets

Kafka

Understanding Message Delivery Semantics

Kafka

Working with Partitions

Kafka

Configuring Producer Settings

Kafka

Configuring Quotas