Working with Partitions

1. Understanding Partition Assignment

Rule Description Detail
Keyed hash(key) % partitions Deterministic placement
Keyless Sticky partitioner batches to one partition Default since 2.4
Explicit Producer sets target partition number Overrides partitioner

Example: Keyed assignment

// Same customer key always routes to the same partition
producer.send(new ProducerRecord<>("orders", "customer-42", payload));

2. Specifying Partition Key

Aspect Effect Detail
Key Present Guarantees ordering per key Co-locates records
Key null Round-robin / sticky distribution No ordering guarantee
Hot Key Skewed key overloads one partition Watch distribution

Example: Record with key

ProducerRecord<String, String> r =
    new ProducerRecord<>("clicks", userId, eventJson);
producer.send(r);

3. Viewing Partition Distribution

Tool Purpose Output
--describe Show leader/replica per partition Layout table
--under-replicated List partitions missing ISR Health check
--at-min-isr Partitions at min ISR threshold Risk indicator

Example: Find under-replicated partitions

kafka-topics.sh --describe --under-replicated-partitions \
  --bootstrap-server localhost:9092

4. Adding Partitions

Aspect Description Caveat
--alter Increase partition count Cannot decrease
Rehashing Existing keys may move partitions Breaks ordering
Consumers Trigger rebalance to pick up new partitions Automatic

Example: Grow partitions

kafka-topics.sh --alter --topic events --partitions 16 \
  --bootstrap-server localhost:9092

5. Understanding Leader Election

Concept Description Detail
Leader Failure Controller elects new leader from ISR Automatic failover
Preferred Leader First replica in the assignment list Balances load
Unclean Election Out-of-sync replica promoted Risks data loss

Example: Trigger preferred election

kafka-leader-election.sh --election-type PREFERRED --all-topic-partitions \
  --bootstrap-server localhost:9092

6. Understanding Log Segments

Concept Description Detail
Base Offset First offset in a segment file name Zero-padded
Active Segment Currently appended segment Retention exempt
Sparse Index Maps offsets to byte positions Fast seeks

Example: Dump a segment

kafka-dump-log.sh --files /var/lib/kafka/events-0/00000000000000000000.log \
  --print-data-log

7. Configuring Preferred Leader Election

Config Description Default
auto.leader.rebalance.enable Auto-restore preferred leaders true
leader.imbalance.
check.interval.seconds		 Imbalance scan interval 300
leader.imbalance.
per.broker.percentage		 Imbalance threshold to trigger 10

Example: Enable auto rebalance

auto.leader.rebalance.enable=true
leader.imbalance.per.broker.percentage=10

8. Configuring Min In-Sync Replicas

Config Description Detail
min.insync.replicas Min ISR required for acks=all write Typically 2
Failure Mode Below min → NotEnoughReplicas Producer blocked
Durability RF=3 + minISR=2 survives 1 broker loss Recommended

Example: Set minimum ISR

kafka-configs.sh --alter --entity-type topics --entity-name payments \
  --add-config min.insync.replicas=2 --bootstrap-server localhost:9092

9. Setting Partition Assignment Strategy

Strategy Behavior Detail
RangeAssignor Assigns contiguous ranges per topic Can skew
RoundRobinAssignor Spreads partitions evenly Balanced
CooperativeStickyAssignor Minimizes movement on rebalance NEW Default-recommended

Example: Cooperative sticky assignor

props.put("partition.assignment.strategy",
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

10. Manual Partition Reassignment

Step Command Detail
Generate --generate Proposes plan
Execute --execute Applies JSON
Verify --verify Checks progress
Throttle --throttle Limits bandwidth

Example: Execute reassignment

kafka-reassign-partitions.sh --reassignment-json-file plan.json \
  --execute --throttle 50000000 --bootstrap-server localhost:9092
kafka-reassign-partitions.sh --reassignment-json-file plan.json \
  --verify --bootstrap-server localhost:9092

Related Cheat Sheets

Kafka

Installing and Configuring Kafka

Kafka

Monitoring Consumer Metrics

Kafka

Using Console Tools

Kafka

Consuming Messages