Understanding Message Compression

1. Understanding Compression Benefits

Benefit Description Detail
Network Less bandwidth used Per batch
Storage Smaller on-disk logs Stored compressed
Throughput More records per request Batch-level

Example: Enable producer compression

props.put("compression.type", "lz4");
props.put("linger.ms", 20); // larger batches compress better

2. Using GZIP Compression

Aspect Value Detail
Ratio Highest Best size reduction
CPU High Slowest
Use Case Bandwidth-constrained Cold paths

Example: GZIP

props.put("compression.type", "gzip");

3. Using Snappy Compression

Aspect Value Detail
Ratio Moderate Balanced
CPU Low Fast
Use Case General throughput Popular

Example: Snappy

props.put("compression.type", "snappy");

4. Using LZ4 Compression

Aspect Value Detail
Ratio Good Solid
CPU Very low Fastest decompress
Use Case Low-latency high throughput Recommended default

Example: LZ4

props.put("compression.type", "lz4");

5. Using ZSTD Compression

Aspect Value Detail
ZSTD NEW Best ratio/CPU balance Since 2.1
Tunable Level 1–22 Configurable
Use Case Maximize savings + speed Modern default

Example: ZSTD

props.put("compression.type", "zstd");

6. Configuring Compression Level

Config Description Detail
compression.
gzip.level		 GZIP level 1–9 Broker/topic
compression.
zstd.level		 ZSTD level 1–22 Higher=smaller
compression.
lz4.level		 LZ4 level Default fast

Example: Tune zstd level

# topic config
compression.type=zstd
compression.zstd.level=6

7. Understanding Compression Ratio

Type Typical Ratio Speed
gzip ~4-5x Slow
zstd ~4-5x Medium
snappy ~2-3x Fast
lz4 ~2-3x Fastest

Example: Monitor compression ratio

# JMX: kafka.producer:type=producer-metrics,client-id=*
# attribute: compression-rate-avg (1.0 = no compression)

8. Comparing Compression Performance

LZ4 / Snappy

  • Low CPU, high speed
  • Moderate ratio
  • Best for latency-sensitive

GZIP / ZSTD

  • Higher CPU
  • Best ratio
  • Best for storage/bandwidth savings

Example: Benchmark settings

kafka-producer-perf-test.sh --topic bench --num-records 1000000 \
  --record-size 1024 --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092 compression.type=zstd

9. Setting Producer Compression

Aspect Description Detail
compression.type Producer-side compression Applied per batch
Scope Whole record batch Not per record
Override Topic config can force re-compress See 12.10

Example: Producer compression

props.put("compression.type", "zstd");
props.put("batch.size", 131072);

10. Broker Compression Handling

Value Behavior Detail
producer Keep producer's codec Default, zero-copy
specific codec Broker re-compresses Extra CPU
uncompressed Store decompressed Rare

Example: Broker compression config

# server.properties — keep producer codec (no recompression)
compression.type=producer

11. Monitoring Compression Metrics

Metric Description Source
compression-rate-avg Avg compression rate Producer JMX
batch-size-avg Avg batch size Producer JMX
BytesInPerSec Broker ingress bytes Broker JMX

Example: Read compression metric

echo "get -b kafka.producer:type=producer-metrics,client-id=p1 compression-rate-avg" \
  | java -jar jmxterm.jar -l localhost:9999

Related Cheat Sheets

Kafka

Installing and Configuring Kafka

Kafka

Configuring Producer Settings

Kafka

Using MirrorMaker for Replication

Kafka

Using Kafka Connect