Capacity Planning and Sizing

1. Estimating Message Throughput

Factor Description Detail
msg/sec Peak event rate Baseline
avg size Bytes per message × rate = MB/s
Headroom 2-3x peak Bursts

Example: Throughput math

100k msg/s × 1 KB = 100 MB/s write
× RF 3 = 300 MB/s replication traffic

2. Calculating Partition Count

Factor Description Detail
Target/Partition 10-50 MB/s each Benchmark
Consumer Parallelism ≥ max consumers Scale ceiling
Avoid Excess Per-partition overhead Don't over-shard

Example: Partition formula

partitions = max(throughput/perPartition, consumerCount)
= max(100/20, 8) = 8 (round up to 12 for headroom)

3. Sizing Broker Hardware

Resource Description Detail
CPU Cores for compression/TLS 8-16+
RAM Page cache key 32-64 GB
Disk NVMe SSD, multiple dirs Throughput

Example: Heap vs page cache

export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"  # leave RAM for page cache

4. Estimating Storage Requirements

Factor Description Detail
Retention Days × daily volume Base
× RF Replication multiplier 3x typical
Overhead Index + buffer 20% Margin

Example: Storage estimate

100 MB/s × 86400 × 7 days × RF3 = ~181 TB
+ 20% overhead = ~217 TB total

5. Planning Network Bandwidth

Factor Description Detail
Ingress Producer writes Base
Replication (RF-1) × ingress Internal
Egress Consumers × fanout Reads

Example: NIC sizing

100 MB/s in + 200 MB/s repl + 300 MB/s out = 600 MB/s
> 10 GbE (1250 MB/s) NIC needed

6. Determining Replication Factor

RF Description Detail
RF=3 Standard production 2 failures tolerated
min.insync=2 Durability floor With acks=all
RF=2 Dev/non-critical Lower cost

Example: Durable topic

kafka-topics.sh --create --topic critical --partitions 12 \
  --replication-factor 3 --config min.insync.replicas=2 \
  --bootstrap-server localhost:9092

7. Sizing Consumer Groups

Factor Description Detail
Consumers ≤ Partitions Extra idle Ceiling
Per-Consumer Rate Processing capacity Benchmark
Lag Target Stay near 0 Add consumers

Example: Group sizing

throughput 100k/s ÷ perConsumer 15k/s = 7 consumers
partitions=12 >= 7 OK (room to scale to 12)

8. Planning for Peak Load

Aspect Description Detail
Peak Multiplier Black Friday spikes 5-10x
Buffer Capacity Absorb bursts Retention
Elastic Scaling Add consumers fast Autoscale

Example: Peak headroom plan

baseline 100 MB/s × 5 peak = 500 MB/s
size cluster for 500, run at 20% normally

9. Configuring Retention Policies

Config Description Detail
retention.ms Time-based Drives storage
retention.bytes Size cap per partition Hard limit
Tiered Long retention cheap Object store

Example: 7-day retention

kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type topics --entity-name events \
  --add-config retention.ms=604800000

10. Monitoring Resource Utilization

Resource Description Detail
Disk % Alert < 20% free Critical
CPU/Network Sustained > 70% Scale signal
Page Cache Read hit ratio Memory health

Example: Disk usage check

df -h /var/lib/kafka/logs
kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe --broker-list 1,2,3

Related Cheat Sheets

Kafka

Managing Cluster Metadata

Kafka

Using Admin API

Kafka

Optimizing Consumer Rebalancing

Kafka

Implementing Log Compaction