Using KRaft Consensus Protocol
1. Understanding KRaft Architecture
| Concept |
Description |
Detail |
| KRaft |
Self-managed metadata via Raft |
No ZooKeeper |
| Controllers |
Raft quorum storing metadata |
Odd count |
| Active Controller |
Raft leader handling writes |
One at a time |
Example: KRaft roles
process.roles=controller # dedicated controller node
| Concept |
Description |
Detail |
| Metadata Log |
__cluster_metadata topic |
Event-sourced |
| Raft Replication |
Quorum-replicated records |
Majority ack |
| Brokers Subscribe |
Replay metadata log |
Eventually consistent |
kafka-metadata-shell.sh --snapshot \
/var/lib/kafka/__cluster_metadata-0/00000000000000000000.log
| Config |
Description |
Default |
| metadata.log.dir |
Metadata log location |
First log.dir |
metadata.log.
max.record.bytes |
Max metadata record |
— |
metadata.max.
retention.ms |
Snapshot/retention |
— |
metadata.log.dir=/data/metadata
4. Setting Controller Quorum Voters
| Config |
Format |
Detail |
| controller.quorum.voters |
id@host:port |
Comma-separated |
| Count |
3 or 5 voters |
Odd for quorum |
| Fault Tolerance |
N voters tolerate (N-1)/2 |
Majority |
Example: Three voters
controller.quorum.voters=1@c1:9093,2@c2:9093,3@c3:9093
| Step |
Command |
Detail |
| random-uuid |
Generate cluster ID |
Once |
| format |
Init log dirs |
Per node |
| --ignore-formatted |
Skip if done |
Idempotent |
bin/kafka-storage.sh format -t $CLUSTER_ID \
-c config/kraft/server.properties
6. Starting KRaft Controller
| Aspect |
Description |
Detail |
| process.roles |
controller |
Dedicated mode |
controller.
listener.names |
CONTROLLER |
Required |
| Start |
kafka-server-start.sh |
Controller config |
Example: Start controller
bin/kafka-server-start.sh config/kraft/controller.properties
7. Starting KRaft Broker
| Aspect |
Description |
Detail |
| process.roles |
broker |
Data node |
controller.
quorum.voters |
Points to controllers |
Required |
| Start |
broker.properties |
Connects to quorum |
Example: Start broker
bin/kafka-server-start.sh config/kraft/broker.properties
8. Using Combined Mode
| Aspect |
Description |
Detail |
| Combined |
broker,controller in one process |
Dev/small |
| Pros |
Fewer nodes |
Simpler |
| Cons |
Not for large prod |
Isolation lost |
Example: Combined roles
process.roles=broker,controller
controller.quorum.voters=1@host:9093
9. Configuring Controller Listener
| Config |
Description |
Detail |
controller.
listener.names |
Listener for controller traffic |
CONTROLLER |
| listeners |
Must include controller port |
:9093 |
| Security |
Map to SSL/SASL if needed |
protocol.map |
Example: Controller listener
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
controller.listener.names=CONTROLLER
10. Migrating from ZooKeeper
| Step |
Action |
Detail |
| Provision |
Deploy KRaft controllers |
Migration mode |
| Enable Migration |
zookeeper.metadata.migration.enable |
Dual-write |
| Finalize |
Remove ZK config |
KRaft-only |
Warning: Kafka 4.0 removes ZooKeeper; migrate to KRaft on 3.x before upgrading to 4.0.
Example: Enable migration
zookeeper.metadata.migration.enable=true
controller.quorum.voters=1@c1:9093,2@c2:9093,3@c3:9093