Installing and Configuring Kafka
1. Downloading Kafka
| Item | Value | Notes |
|---|---|---|
| Source | kafka.apache.org/downloads |
Binary tarball |
| Latest Stable | Kafka 3.9 / 4.0 NEW | Scala 2.13 build |
| Package | kafka_2.13-4.0.0.tgz |
Includes broker + tools |
Example: Download and extract
wget https://downloads.apache.org/kafka/4.0.0/kafka_2.13-4.0.0.tgz
tar -xzf kafka_2.13-4.0.0.tgz
cd kafka_2.13-4.0.02. Setting Up Java Requirements
| Requirement | Value | Notes |
|---|---|---|
| JDK | Java 17+ NEW | Required by Kafka 4.0 |
| Broker JVM | Java 17 or 21 LTS | Recommended for prod |
| Env Var | JAVA_HOME |
Points to JDK root |
Example: Verify Java
java -version # openjdk version "21.0.2"
export JAVA_HOME=/usr/lib/jvm/java-21-openjdk3. Configuring Broker Properties
| Property | Purpose | Example |
|---|---|---|
| node.id | Unique broker/controller identifier | 1 |
| process.roles | Roles in KRaft mode | broker,controller |
| log.dirs | Data directories | /var/lib/kafka |
| listeners | Bind addresses | PLAINTEXT://:9092 |
Example: Core server.properties
process.roles=broker,controller
node.id=1
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
log.dirs=/var/lib/kafka/data
num.partitions=34. Setting Broker ID
| Property | Description | Constraint |
|---|---|---|
| node.id | KRaft identifier for broker and controller | Unique integer |
| broker.id | Legacy ZooKeeper-mode identifier | LEGACY |
| Auto-generation | Reserved IDs start at reserved.broker.max.id+1 |
Avoid in prod |
5. Configuring Log Directories
| Property | Description | Tip |
|---|---|---|
| log.dirs | Comma-separated data directories | One per disk for IO spread |
| log.dir | Single directory fallback | Overridden by log.dirs |
| Placement | Use dedicated, fast disks (XFS) | Avoid root volume |
6. Configuring Listeners
| Property | Purpose | Example |
|---|---|---|
| listeners | Interfaces the broker binds to | PLAINTEXT://:9092 |
| advertised.listeners | Addresses clients use to connect | PLAINTEXT://host:9092 |
| listener.security. protocol.map |
Maps listener names to protocols | INTERNAL:SSL |
Example: Internal and external listeners
listeners=INTERNAL://:9092,EXTERNAL://:9094
advertised.listeners=INTERNAL://kafka1:9092,EXTERNAL://public.example.com:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
inter.broker.listener.name=INTERNAL7. Setting Up Cluster ID
| Step | Command | Detail |
|---|---|---|
| Generate | kafka-storage.sh random-uuid |
Base64 UUID |
| Format | kafka-storage.sh format |
Writes meta.properties |
| Reuse | Same cluster ID on all nodes | Identifies cluster |
Example: Generate and format storage
KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-uuid)
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID \
-c config/kraft/server.properties8. Starting ZooKeeper Server
| Item | Value | Status |
|---|---|---|
| Script | zookeeper-server-start.sh |
LEGACY |
| Config | config/zookeeper.properties |
Pre-4.0 only |
| Port | 2181 |
Client port |
9. Starting Kafka Broker
| Item | Value | Notes |
|---|---|---|
| Script | kafka-server-start.sh |
Starts broker |
| Daemon Flag | -daemon |
Run in background |
| Heap | KAFKA_HEAP_OPTS |
e.g. -Xmx6g |
Example: Start broker
export KAFKA_HEAP_OPTS="-Xmx6g -Xms6g"
bin/kafka-server-start.sh -daemon config/kraft/server.properties10. Using KRaft Mode
| Step | Action | Notes |
|---|---|---|
| Config | Set process.roles and quorum voters |
No ZooKeeper |
| Format | Run kafka-storage.sh format |
Once per node |
| Start | Launch with KRaft config | Self-managed metadata |
Example: Single-node KRaft startup
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) \
-c config/kraft/server.properties --standalone
bin/kafka-server-start.sh config/kraft/server.properties11. Verifying Installation
| Check | Command | Expected |
|---|---|---|
| Broker API | kafka-broker-api-versions.sh |
Lists API versions |
| Create Topic | kafka-topics.sh --create |
Topic created |
| Roundtrip | Console producer + consumer | Message echoed |