Optimizing Broker Performance

1. Tuning Number of Partitions

Aspect Description Detail
Parallelism More partitions = more Consumer scale
Overhead File handles, memory Too many costs
Rule Throughput / per-partition Estimate

Example: Partition count

partitions = max(target/producerRate, target/consumerRate)

2. Optimizing Replica Fetcher Threads

Config Description Default
num.replica.fetchers Threads per broker 1
Higher Faster replication More CPU
Use High partition count Catch-up

Example: Replica fetchers

num.replica.fetchers=4

3. Configuring Network Threads

Config Description Default
num.network.threads Accept/respond threads 3
Scale With connections/cores ~cores
Idle Metric NetworkProcessorAvgIdle >0.3 healthy

Example: Network threads

num.network.threads=8

4. Tuning I/O Threads

Config Description Default
num.io.threads Disk request handlers 8
Rule ≥ number of disks Saturate IO
Idle Metric RequestHandlerAvgIdle >0.3 healthy

Example: IO threads

num.io.threads=16

5. Optimizing Log Segment Size

Aspect Description Detail
Larger Segments Fewer files Less overhead
Smaller Segments Faster cleanup Granular retention
log.segment.bytes Per-segment size 1GB default

Example: Segment size

log.segment.bytes=1073741824

6. Configuring Zero-Copy Transfer

Aspect Description Detail
sendfile Kernel-level copy No user buffer
Plaintext Zero-copy works Fast
TLS Disables zero-copy CPU cost

Example: Zero-copy note

// PLAINTEXT fetch uses sendfile(); SSL routes through user space

7. Tuning Flush Settings

Config Description Default
log.flush.interval.
messages		 Force fsync after N Long.MAX
OS Default Rely on page cache Recommended
Durability Replication, not fsync RF ≥ 3

Example: Default flush (OS-managed)

# Leave defaults; durability comes from replication
# log.flush.interval.messages=10000  (usually not needed)

8. Optimizing Page Cache

Aspect Description Detail
OS Cache Kafka relies on it Hot reads
Heap Keep JVM small ~6GB, rest to cache
RAM More = better cache Read perf

Example: JVM heap sizing

export KAFKA_HEAP_OPTS="-Xmx6g -Xms6g"  # leave RAM for page cache

9. Setting Socket Buffer Sizes

Config Description Default
socket.send.
buffer.bytes		 TCP send buffer 102400
socket.receive.
buffer.bytes		 TCP receive buffer 102400
High BDP Increase for WAN Throughput

Example: Socket buffers

socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576

10. Monitoring Broker Metrics

Metric Description Watch
UnderReplicatedPartitions Replication lag =0
RequestHandlerAvgIdle IO thread headroom >0.3
BytesInPerSec Ingress throughput Trend

Example: Broker JMX

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

Related Cheat Sheets

Kafka

Installing and Configuring Kafka

Kafka

Implementing Custom Interceptors

Kafka

Managing Offsets

Kafka

Implementing Security