Managing Cluster Operations
1. Adding Brokers to Cluster
| Step |
Description |
Detail |
| Unique ID |
New node.id |
Distinct |
| Start |
Joins cluster |
No partitions yet |
| Reassign |
Move partitions to it |
Rebalance |
Example: New broker config
node.id=4
controller.quorum.voters=1@c1:9093,2@c2:9093,3@c3:9093
2. Removing Brokers from Cluster
| Step |
Description |
Detail |
| Drain |
Move partitions off |
Reassign first |
| Verify |
No leaders remain |
Check |
| Shutdown |
Stop broker |
Safe |
Example: Reassign off broker 4
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
--reassignment-json-file drain-broker4.json --execute
3. Expanding Cluster
| Aspect |
Description |
Detail |
| Add Capacity |
More brokers |
Horizontal |
| Rebalance |
Spread existing data |
Even load |
| Throttle |
Limit move bandwidth |
No disruption |
Example: Throttled reassignment
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
--reassignment-json-file plan.json --execute --throttle 50000000
4. Balancing Partition Load
| Aspect |
Description |
Detail |
| Generate Plan |
--generate proposes |
Even spread |
| Cruise Control |
Auto-balancing tool |
Continuous |
| Preferred Leader |
Restore leadership balance |
Election |
Example: Generate balance plan
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
--topics-to-move-json-file topics.json \
--broker-list "1,2,3,4" --generate
5. Rolling Restart
| Step |
Description |
Detail |
| One at a Time |
Restart sequentially |
Availability |
| Wait URP=0 |
Before next broker |
Safe |
| controlled.shutdown |
Graceful leader move |
Default on |
Example: Wait for URP=0
# Restart broker, then verify before continuing
kafka-topics.sh --bootstrap-server localhost:9092 \
--describe --under-replicated-partitions
6. Upgrading Kafka Version
| Step |
Description |
Detail |
| Rolling |
One broker at a time |
No downtime |
inter.broker.
protocol.version |
Pin during upgrade |
Compat |
| Bump After |
Raise version last |
Finalize |
Example: Pin protocol version
inter.broker.protocol.version=3.8
# After all brokers upgraded, set to 3.9 and restart
7. Migrating to KRaft Mode
| Step |
Description |
Detail |
| Deploy Controllers |
KRaft quorum |
Migration mode |
| Dual-Write |
ZK + KRaft |
Transition |
| Finalize |
Drop ZK |
KRaft-only |
Warning: Migrate to KRaft on 3.x before upgrading to 4.0, which removes ZooKeeper entirely.
Example: Migration flag
zookeeper.metadata.migration.enable=true
8. Handling Broker Failures
| Aspect |
Description |
Detail |
| Auto Failover |
ISR replica promoted |
No data loss |
| Recovery |
Restart, catch up |
Rejoin ISR |
| Replacement |
New broker + reassign |
Permanent loss |
Example: Check leadership after failure
kafka-topics.sh --bootstrap-server localhost:9092 \
--describe --under-replicated-partitions
9. Managing Disk Space
| Aspect |
Description |
Detail |
| Retention |
Reduce to free space |
time/bytes |
| Add Disk |
Extra log.dirs |
Capacity |
| Delete Records |
Truncate old data |
deleteRecords |
Example: Shrink retention
kafka-configs.sh --bootstrap-server localhost:9092 --alter \
--entity-type topics --entity-name bulky \
--add-config retention.ms=43200000
10. Configuring Rack Awareness
| Config |
Description |
Detail |
| broker.rack |
AZ/rack identifier |
Per broker |
| Spread |
Replicas across racks |
Fault isolation |
| Rack-Aware Fetch |
Consume from local replica |
client.rack |