Understanding Kafka Architecture

1. Understanding Broker Role

Concept Description Key Detail
Broker A single Kafka server that stores data and serves client requests Identified by unique broker.id
Cluster Group of brokers working together for scalability and fault tolerance Shares one cluster ID
Controller One broker elected to manage partition leadership and metadata Single active controller per cluster
Bootstrap Server Initial broker(s) clients connect to for cluster discovery host:9092

Example: Connecting to brokers

Properties props = new Properties();
// Comma-separated list for failover; client discovers full cluster
props.put("bootstrap.servers", "broker1:9092,broker2:9092,broker3:9092");

2. Understanding Topics and Partitions

Concept Description Property
Topic Named logical stream of records (category/feed name) Multi-subscriber
Partition Ordered, immutable append-only log; unit of parallelism num.partitions
Partition Key Determines which partition a record lands in via hashing Same key → same partition
Ordering Guaranteed only within a single partition, not across topic Per-partition

Example: Creating a topic with partitions

kafka-topics.sh --create --topic orders \
  --partitions 6 --replication-factor 3 \
  --bootstrap-server localhost:9092

3. Understanding Producers and Consumers

Client Role Key Behavior
Producer Publishes records to topics Chooses partition, batches, retries
Consumer Reads records from topics in order per partition Tracks offsets, pulls via poll
Pull Model Consumers request data rather than brokers pushing Controls consumption rate

Example: Minimal produce and consume

producer.send(new ProducerRecord<>("orders", "key1", "payload"));

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> r : records)
    System.out.printf("%s @ %d%n", r.value(), r.offset());

4. Understanding Consumer Groups

Concept Description Rule
Consumer Group Set of consumers sharing a group.id to split work Scales horizontally
Partition Ownership Each partition is consumed by exactly one member of a group 1 partition → 1 consumer
Parallelism Limit Max active consumers = partition count Extra consumers idle
Broadcast Different groups each receive all messages independently Pub/sub semantics

Example: Assigning a consumer group

props.put("group.id", "order-processors");
// Three instances with same group.id share 6 partitions (2 each)
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));

5. Understanding Replication

Term Description Detail
Replication Factor Number of copies of each partition across brokers RF=3 tolerates 2 failures
Leader Replica handling all reads and writes for a partition One per partition
Follower Replica that passively copies the leader's log Promoted on failure
ISR In-Sync Replicas caught up with the leader Eligible for election

Example: Inspecting replication

kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
# Topic: orders Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3

6. Understanding Log Segments

Concept Description Config
Segment Partition log split into fixed-size files on disk segment.bytes
Active Segment Current segment receiving new writes Never deleted
Index Files .index / .timeindex for fast offset lookup Per segment
Roll New segment created on size or time threshold segment.ms

Example: On-disk segment files

ls /var/lib/kafka/orders-0/
# 00000000000000000000.log
# 00000000000000000000.index
# 00000000000000000000.timeindex

7. Understanding Offsets

Term Description Detail
Offset Sequential 64-bit ID of a record within a partition Monotonic, never reused
Committed Offset Last position a group successfully processed Stored in __consumer_offsets
Log End Offset Offset of the next record to be appended LEO
Lag LEO minus committed offset Backlog indicator

Example: Reading current offset

for (ConsumerRecord<String, String> r : records) {
    process(r);
    // offset of next record to read after this one
    long next = r.offset() + 1;
}

8. Understanding Message Flow

Producer ──▶ [Partitioner] ──▶ Leader Partition (Broker 1)
                                   │  replicate
                          ┌────────┴────────┐
                          ▼                 ▼
                   Follower (Broker 2)  Follower (Broker 3)
                          │
                          ▼
                   Consumer Group (poll)
Step Action
1. Produce Record serialized, partitioned, batched, sent to leader
2. Replicate Followers fetch and append to stay in ISR
3. Ack Leader acknowledges per acks setting
4. Consume Consumer polls leader, commits offset after processing

Example: End-to-end flow with ack

props.put("acks", "all"); // wait for all ISR before ack
RecordMetadata md = producer.send(record).get();
System.out.printf("partition=%d offset=%d%n", md.partition(), md.offset());

9. Understanding ZooKeeper Role

Responsibility Description Status
Controller Election Picks the active controller broker LEGACY
Metadata Store Holds topic, partition, ACL, and config state Replaced by KRaft
Membership Tracks live brokers via ephemeral znodes Removed in 4.0
Warning: ZooKeeper mode is removed in Kafka 4.0+. Use KRaft for all new deployments.

Example: Legacy ZooKeeper connection

# server.properties (legacy mode, pre-4.0)
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181

10. Understanding KRaft Architecture

Concept Description Detail
KRaft DEFAULT Kafka Raft metadata mode; no ZooKeeper dependency Default since 3.3+
Controller Quorum Raft-based group of controllers replicating metadata controller.quorum.voters
Metadata Log Internal __cluster_metadata topic as source of truth Event-sourced
Combined Mode One process acts as both broker and controller Dev/small clusters

Example: KRaft controller config

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093
controller.listener.names=CONTROLLER

Related Cheat Sheets

Kafka

Understanding Replication

Kafka

Producing Messages

Kafka

Configuring Consumer Settings

Kafka

Consuming Messages