Skip to main content

Home
Datastructures-algorithms
Frontend
Backend
Database
Messaging
DevOps

Datastructures Algorithms
Algorithm Patterns

HTML
CSS
SCSS
Javascript
Typescript
Web Apis
Javascript Polyfills
React
React State Management
RXJS
Accessibility
Low Level Design
High Level Design

Advanced Java
API Gateway
Authentication Authorization
Core Java
Distributed Systems
Expressjs
GRAPHQL
GRPC
High Level Design
Low Level Design
Microservices Design Patterns
Microservices
NODEJS
Prisma
REST API
Spring Boot
Websockets

Database Fundamentals
Mongodb
Pinecone
Postgresql
Redis

Kafka

Helm Charts
Kubernetes

Home
Messaging
Kafka

Kafka Roadmap

56 sections • 1174 topics

1

Understanding Kafka Architecture

1. Understanding Broker Role
Example: Connecting to brokers
2. Understanding Topics and Partitions
Example: Creating a topic with partitions
3. Understanding Producers and Consumers
Example: Minimal produce and consume
4. Understanding Consumer Groups
Example: Assigning a consumer group
5. Understanding Replication
Example: Inspecting replication
6. Understanding Log Segments
Example: On-disk segment files
7. Understanding Offsets
Example: Reading current offset
8. Understanding Message Flow
Example: End-to-end flow with ack
9. Understanding ZooKeeper Role
Example: Legacy ZooKeeper connection
10. Understanding KRaft Architecture
Example: KRaft controller config

2

Installing and Configuring Kafka

1. Downloading Kafka
Example: Download and extract
2. Setting Up Java Requirements
Example: Verify Java
3. Configuring Broker Properties
Example: Core server.properties
4. Setting Broker ID
Example: Set node id
5. Configuring Log Directories
Example: Multiple log dirs
6. Configuring Listeners
Example: Internal and external listeners
7. Setting Up Cluster ID
Example: Generate and format storage
8. Starting ZooKeeper Server
Example: Start ZooKeeper (legacy)
9. Starting Kafka Broker
Example: Start broker
10. Using KRaft Mode
Example: Single-node KRaft startup
11. Verifying Installation
Example: Smoke test

3

Managing Topics

1. Creating Topics
Example: Create with configs
2. Listing Topics
Example: List topics
3. Describing Topics
Example: Describe a topic
4. Setting Partition Count
Example: Add partitions
5. Setting Replication Factor
Example: Increase RF via reassignment
6. Altering Topic Configuration
Example: Alter retention
7. Deleting Topics
Example: Delete a topic
8. Configuring Topic Retention
Example: Time + size retention
9. Setting Cleanup Policy
Example: Enable compaction
10. Configuring Segment Settings
Example: Smaller segments for faster compaction

4

Working with Partitions

1. Understanding Partition Assignment
Example: Keyed assignment
2. Specifying Partition Key
Example: Record with key
3. Viewing Partition Distribution
Example: Find under-replicated partitions
4. Adding Partitions
Example: Grow partitions
5. Understanding Leader Election
Example: Trigger preferred election
6. Understanding Log Segments
Example: Dump a segment
7. Configuring Preferred Leader Election
Example: Enable auto rebalance
8. Configuring Min In-Sync Replicas
Example: Set minimum ISR
9. Setting Partition Assignment Strategy
Example: Cooperative sticky assignor
10. Manual Partition Reassignment
Example: Execute reassignment

5

Producing Messages

1. Creating Producer Instance
Example: Construct a producer
2. Setting Producer Properties
Example: Durable producer config
3. Configuring Bootstrap Servers
Example: Multiple bootstrap brokers
4. Configuring Serializers
Example: Avro value serializer
5. Sending Messages
Example: Send a record
6. Using Synchronous Send
Example: Synchronous send
7. Using Asynchronous Send
Example: Async with callback
8. Setting Message Key
Example: Keyed record
9. Specifying Target Partition
Example: Send to specific partition
10. Adding Message Headers
Example: Add headers
11. Configuring Acknowledgments
Example: Strongest durability
12. Flushing Producer Buffer
Example: Flush before exit

6

Configuring Producer Settings

1. Setting Batch Size
Example: Larger batches
2. Configuring Linger Time
Example: Batch for throughput
3. Setting Buffer Memory
Example: Increase buffer
4. Configuring Compression
Example: Enable zstd
5. Setting Max Request Size
Example: Allow larger messages
6. Setting Request Timeout
Example: Set request timeout
7. Configuring Delivery Timeout
Example: Cap total delivery time
8. Configuring Retries
Example: Retry config
9. Setting Idempotence
Example: Idempotent producer
10. Configuring Max In-Flight Requests
Example: Set in-flight limit

7

Consuming Messages

1. Creating Consumer Instance
Example: Construct a consumer
2. Setting Consumer Properties
Example: Manual-commit config
3. Configuring Bootstrap Servers
Example: Bootstrap servers
4. Configuring Deserializers
Example: Avro deserializer
5. Setting Consumer Group
Example: Static membership
6. Subscribing to Topics
Example: Subscribe to topics
7. Polling Messages
Example: Poll loop
8. Processing Records
Example: Access record fields
9. Using Manual Assignment
Example: Manual assign
10. Seeking to Position
Example: Seek to offset
11. Closing Consumer
Example: Graceful shutdown

8

Configuring Consumer Settings

1. Setting Fetch Size
Example: Batch-oriented fetch
2. Configuring Fetch Wait Time
Example: Reduce fetch latency
3. Setting Max Poll Records
Example: Limit poll batch
4. Setting Max Partition Fetch Bytes
Example: Increase per-partition fetch
5. Configuring Partition Assignment Strategy
Example: Set strategy
6. Setting Auto Offset Reset
Example: Process from beginning
7. Configuring Isolation Level
Example: Read committed only
8. Setting Session Timeout
Example: Tune session timeout
9. Setting Heartbeat Interval
Example: Heartbeat config
10. Configuring Max Poll Interval
Example: Allow longer processing
11. Configuring Check CRCs
Example: CRC check setting
12. Setting Client ID
Example: Set client id

9

Managing Consumer Groups

1. Understanding Group Coordinator
Example: Find coordinator
2. Understanding Rebalancing
Example: Rebalance listener
3. Listing Consumer Groups
Example: List groups
4. Describing Consumer Groups
Example: Describe a group
5. Viewing Consumer Lag
Example: Check lag
6. Resetting Offsets
Example: Reset to earliest
7. Deleting Consumer Groups
Example: Delete a group
8. Configuring Session Timeout
Example: Broker bounds
9. Setting Heartbeat Interval
Example: Align heartbeat
10. Configuring Max Poll Interval
Example: Prevent eviction during heavy work

10

Managing Offsets

1. Understanding Offset Commits
Example: Commit after processing
2. Enabling Auto Commit
Example: Enable auto commit
3. Setting Auto Commit Interval
Example: Frequent auto commit
4. Committing Synchronously
Example: Synchronous commit
5. Committing Asynchronously
Example: Async with final sync
6. Committing Specific Offsets
Example: Commit explicit offsets
7. Storing Offsets Externally
Example: Restore offset from DB
8. Resetting to Earliest
Example: Seek to beginning
9. Resetting to Latest
Example: Skip to latest
10. Resetting to Specific Offset
Example: Reset to offset 1000
11. Resetting by Timestamp
Example: Reset by datetime

11

Working with Message Serialization

1. Understanding Serialization Concept
Example: Serializer interface
2. Using String Serializer
Example: String serializer
3. Using Byte Array Serializer
Example: Byte array producer
4. Using Integer Serializer
Example: Integer key
5. Using Long Serializer
Example: Long timestamp value
6. Using JSON Serialization
Example: Custom JSON serializer
7. Using Avro Serialization
Example: Avro producer
8. Using Protobuf Serialization
Example: Protobuf serializer
9. Creating Custom Serializers
Example: Register custom serializer
10. Handling Serialization Errors
Example: Safe deserialization wrapper
11. Configuring Schema Registry
Example: Registry client config

12

Understanding Message Compression

1. Understanding Compression Benefits
Example: Enable producer compression
2. Using GZIP Compression
Example: GZIP
3. Using Snappy Compression
Example: Snappy
4. Using LZ4 Compression
Example: LZ4
5. Using ZSTD Compression
Example: ZSTD
6. Configuring Compression Level
Example: Tune zstd level
7. Understanding Compression Ratio
Example: Monitor compression ratio
8. Comparing Compression Performance
LZ4 / Snappy
GZIP / ZSTD
Example: Benchmark settings
9. Setting Producer Compression
Example: Producer compression
10. Broker Compression Handling
Example: Broker compression config
11. Monitoring Compression Metrics
Example: Read compression metric

13

Working with Message Headers

1. Understanding Headers Concept
Example: Headers interface
2. Adding Headers to Producer
Example: Add producer headers
3. Reading Headers in Consumer
Example: Read a header
4. Setting Custom Headers
Example: Multiple custom headers
5. Using Headers for Routing
Example: Route by header
6. Storing Metadata in Headers
Example: Retry count header
7. Filtering by Headers
Example: Filter by header
8. Understanding Header Serialization
Example: Encode/decode header value
9. Propagating Headers in Streams
Example: Headers in processor
10. Adding Trace IDs
Example: Inject trace id

14

Understanding Message Timestamps

1. Understanding Timestamp Types
Example: Read record timestamp
2. Using Create Time
Example: Explicit create time
3. Using Log Append Time
Example: Set log append time
4. Configuring Timestamp Type
Example: Broker default
5. Setting Producer Timestamp
Example: Backdated event
6. Understanding Timestamp Ordering
Example: Detect out-of-order
7. Using Timestamps for Retention
Example: Retention with append time
8. Filtering by Timestamp
Example: Seek by timestamp
9. Configuring Timestamp Difference
Example: Limit timestamp skew
10. Resetting Offsets by Timestamp
Example: Reset group to datetime

15

Understanding Message Ordering

1. Understanding Ordering Guarantees
Example: Per-key ordering
2. Preserving Order with Single Partition
Example: Single-partition topic
3. Using Message Keys for Ordering
Example: Key-based ordering
4. Configuring Max In-Flight Requests
Example: Safe in-flight + idempotence
5. Enabling Idempotent Producer
Example: Idempotent producer
6. Understanding Ordering Across Partitions
Example: Reorder by timestamp buffer
7. Implementing Total Ordering
Example: Embed sequence number
8. Handling Out-of-Order Messages
Example: Streams grace period
9. Using Timestamps for Ordering
Example: Custom timestamp extractor
10. Configuring Retries for Ordering
Example: Order-preserving retries

16

Understanding Replication

1. Understanding Replication Concept
Example: Topic with RF=3
2. Understanding Leader and Followers
Example: Identify leader
3. Configuring Replication Factor
Example: Broker default RF
4. Understanding In-Sync Replicas
Example: Check ISR
5. Configuring Replica Lag
Example: Tune replica lag window
6. Understanding Leader Election
Example: Disable unclean election
7. Understanding Replica High Watermark
Example: HW limits reads
8. Understanding Log End Offset
Example: Inspect LEO via JMX
9. Configuring Preferred Leader
Example: Run preferred election
10. Understanding Replica Fetcher
Example: More fetcher threads
11. Manual Replica Reassignment
Example: Reassign replicas

17

Implementing Log Compaction

1. Understanding Compaction Concept
Example: Compacted topic concept
2. Enabling Log Compaction
Example: Enable compaction
3. Understanding Compaction Process
Example: Cleaner status in logs
4. Understanding Tombstone Messages
Example: Send a tombstone
5. Configuring Compaction Lag
Example: Set compaction lag
6. Setting Delete Retention
Example: Extend delete retention
7. Configuring Segment Size
Example: Smaller segments for compaction
8. Setting Min Cleanable Ratio
Example: Aggressive cleaning
9. Configuring Compaction Thread
Example: More cleaner threads
10. Setting Compaction Buffer
Example: Larger dedupe buffer
11. Monitoring Compaction Progress
Example: Cleaner JMX bean

18

Understanding Message Delivery Semantics

1. Understanding Delivery Guarantees
Example: At-least-once
2. Understanding At-Most-Once Delivery
Example: At-most-once
3. Understanding At-Least-Once Delivery
Example: At-least-once with idempotent sink
4. Implementing Exactly-Once Semantics
Example: Streams exactly-once
5. Configuring Producer Idempotence
Example: Idempotent config
6. Using Transactional Producer
Example: Transactional send
7. Setting Consumer Isolation Level
Example: Read committed
8. Understanding Duplicate Messages
Example: Dedup by key+offset
9. Handling Message Ordering
Example: Ordered EOS pipeline
10. Implementing Idempotent Consumer
Example: Atomic DB upsert + offset
11. Understanding Message Loss Scenarios
Example: No-loss producer/topic

19

Implementing Transactions

1. Understanding Transactional Concept
Example: Transactional pipeline shape
2. Enabling Transactional Producer
Example: Configure transactional id
3. Initializing Transactions
Example: Init transactions
4. Beginning Transaction
Example: Begin a transaction
5. Sending Transactional Messages
Example: Multi-topic atomic write
6. Sending Offsets in Transaction
Example: Offsets within transaction
7. Committing Transaction
Example: Commit
8. Aborting Transaction
Example: Abort on failure
9. Configuring Transaction Timeout
Example: Set transaction timeout
10. Setting Read Committed
Example: Consumer read committed
11. Understanding Exactly-Once Semantics
Example: Streams EOS

20

Implementing Retry Mechanisms

1. Understanding Retry Behavior
Example: Retry config
2. Configuring Retry Count
Example: Limit retries
3. Setting Retry Backoff
Example: Backoff config
4. Configuring Delivery Timeout
Example: Set delivery deadline
5. Understanding Retriable Exceptions
Example: Check retriable
6. Handling Non-Retriable Errors
Example: Route non-retriable to DLQ
7. Implementing Exponential Backoff
Example: Consumer-side backoff
8. Using Circuit Breaker Pattern
Example: Circuit breaker around sink
9. Configuring Request Timeout
Example: Request timeout
10. Implementing Consumer Retry Logic
Example: Retry-topic pattern

21

Configuring Broker Settings

1. Setting Broker ID
Example: Broker id
2. Configuring Listeners
Example: Listener config
3. Setting Log Directories
Example: Multiple disks
4. Configuring Number of Threads
Example: Thread tuning
5. Setting Socket Buffer Sizes
Example: Larger socket buffers
6. Configuring Max Message Size
Example: Allow large messages
7. Setting Default Replication Factor
Example: Replication defaults
8. Configuring Min ISR
Example: Default min ISR
9. Setting Unclean Leader Election
Example: Keep unclean disabled
10. Configuring Log Retention
Example: Retention defaults
11. Setting Log Segment Size
Example: Segment size
12. Configuring Log Cleanup
Example: Cleanup policy

22

Using KRaft Consensus Protocol

1. Understanding KRaft Architecture
Example: KRaft roles
2. Understanding Metadata Replication
Example: Inspect metadata log
3. Configuring Metadata Log
Example: Dedicated metadata dir
4. Setting Controller Quorum Voters
Example: Three voters
5. Formatting Storage
Example: Format storage
6. Starting KRaft Controller
Example: Start controller
7. Starting KRaft Broker
Example: Start broker
8. Using Combined Mode
Example: Combined roles
9. Configuring Controller Listener
Example: Controller listener
10. Migrating from ZooKeeper
Example: Enable migration

23

Managing Cluster Metadata

1. Understanding Metadata Topics
Example: List internal topics
2. Viewing Cluster Metadata
Example: Metadata shell
3. Describing Cluster Configuration
Example: Describe cluster config
4. Managing Broker Configuration
Example: Dynamic broker change
5. Understanding Controller Role
Example: Find active controller
6. Monitoring Controller Metrics
Example: JMX object name
7. Handling Metadata Updates
Example: Replication lag
8. Configuring Metadata Max Age
Example: Client metadata age
9. Understanding Partition Metadata
Example: Describe partition metadata
10. Debugging Metadata Issues
Example: Dump metadata records

24

Using Kafka Streams

1. Understanding Streams Concept
Example: Streams dependency
2. Creating Streams Application
Example: App skeleton
3. Building Stream Topology
Example: Describe topology
4. Transforming Streams
Example: Map and filter
5. Aggregating Streams
Example: Count per key
6. Joining Streams
Example: Stream-table join
7. Windowing Operations
Example: Windowed count
8. Using State Stores
Example: Query a store
9. Processing Records
Example: Peek for logging
10. Using Processors API
Example: Processor API
11. Starting Streams Application
Example: Start with hook
12. Handling Streams Exceptions
Example: Replace failed thread

25

Configuring Kafka Streams

1. Setting Application ID
Example: Application id
2. Configuring Bootstrap Servers
Example: Bootstrap servers
3. Setting Default Serdes
Example: Default serdes
4. Configuring Commit Interval
Example: Commit interval
5. Setting Cache Size
Example: Cache size
6. Configuring State Directory
Example: State dir
7. Setting Processing Guarantee
Example: Exactly-once
8. Configuring Replication Factor
Example: Internal topic RF
9. Setting Number of Threads
Example: Stream threads
10. Configuring Window Store Retention
Example: Window retention

26

Working with Time-Based Processing

1. Understanding Time Semantics
Example: Time concept
2. Understanding Event Time
Example: Default extractor
3. Understanding Processing Time
Example: Wall-clock extractor
4. Understanding Ingestion Time
Example: Set ingestion time
5. Configuring Timestamp Extractors
Example: Per-stream extractor
6. Implementing Custom Timestampers
Example: Custom extractor
7. Using Tumbling Windows
Example: Tumbling window
8. Using Hopping Windows
Example: Hopping window
9. Using Sliding Windows
Example: Sliding window
10. Using Session Windows
Example: Session window
11. Configuring Grace Period
Example: Window with grace
12. Handling Out-of-Order Events
Example: Suppress until close

27

Implementing Stream Joins

1. Understanding Join Types
Example: Join overview
2. Using Stream-Stream Join
Example: Stream-stream join
3. Using Stream-Table Join
Example: Enrich orders
4. Using Table-Table Join
Example: Table-table join
5. Implementing Inner Join
Example: Inner join
6. Implementing Left Join
Example: Left join with default
7. Implementing Outer Join
Example: Outer join
8. Configuring Join Windows
Example: Asymmetric window
9. Setting Join Retention
Example: Join store config
10. Understanding Join Semantics
Example: GlobalKTable join
11. Optimizing Join Performance
Example: Avoid repartition

28

Using Kafka Connect

1. Understanding Connect Framework
Example: Connect roles
2. Running Standalone Mode
Example: Start standalone
3. Running Distributed Mode
Example: Start distributed
4. Creating Source Connectors
Example: JDBC source
5. Creating Sink Connectors
Example: S3 sink
6. Configuring Connector Properties
Example: SMT transform
7. Managing Connectors via REST API
Example: Create via REST
8. Listing Connectors
Example: List connectors
9. Checking Connector Status
Example: Status check
10. Pausing and Resuming Connectors
Example: Pause connector
11. Deleting Connectors
Example: Delete connector

29

Configuring Kafka Connect

1. Setting Bootstrap Servers
Example: Bootstrap
2. Configuring Key and Value Converters
Example: Converters
3. Setting Offset Storage
Example: Offset topic
4. Configuring Config Storage
Example: Config topic
5. Setting Status Storage
Example: Status topic
6. Configuring Replication Factor
Example: RF for all internal topics
7. Setting Group ID
Example: Group id
8. Configuring Plugin Path
Example: Plugin path
9. Setting REST Port
Example: REST listener
10. Configuring Internal Topics
Example: Pre-create config topic

30

Using Schema Registry

1. Understanding Schema Registry Role
Example: Wire format
2. Installing Schema Registry
Example: Start registry
3. Configuring Schema Registry URL
Example: Client config
4. Registering Schemas
Example: Register schema
5. Retrieving Schemas
Example: Get latest
6. Listing Subjects
Example: List subjects
7. Deleting Schema Versions
Example: Delete a version
8. Setting Compatibility Mode
Example: Set subject compatibility
9. Checking Compatibility
Example: Check compatibility
10. Exporting Schemas
Example: Export all subjects

31

Managing Schema Evolution

1. Understanding Schema Compatibility
Example: Field with default
2. Implementing Backward Compatibility
Example: Backward-safe add
3. Implementing Forward Compatibility
Example: Forward-safe add
4. Using Full Compatibility
Example: Full mode
5. Using Transitive Compatibility
Example: Transitive mode
6. Configuring Compatibility Level
Example: Global default
7. Handling Schema Evolution Errors
Example: Pre-deploy check
8. Versioning Schemas
Example: List versions
9. Using Schema References
Example: Schema with reference
10. Migrating Schema Formats
Example: Set schema type

32

Implementing Custom Partitioners

1. Understanding Partitioner Concept
Example: Interface
2. Creating Custom Partitioner
Example: Class skeleton
3. Implementing partition() Method
Example: partition()
4. Configuring Partitioner Class
Example: Register partitioner
5. Using Hash-Based Partitioning
Example: Murmur2 hash
6. Implementing Round-Robin Partitioning
Example: Round-robin
7. Creating Sticky Partitioner
Example: Built-in sticky behavior
8. Implementing Range Partitioner
Example: Range mapping
9. Using Composite Key Partitioning
Example: Tenant prefix routing
10. Handling Null Keys
Example: Null-key fallback

33

Implementing Custom Interceptors

1. Understanding Interceptor Concept
Example: Interfaces
2. Creating Producer Interceptor
Example: Producer interceptor
3. Creating Consumer Interceptor
Example: Consumer interceptor
4. Implementing onSend() Method
Example: Add header in onSend
5. Implementing onConsume() Method
Example: Count in onConsume
6. Configuring Interceptor Chain
Example: Register chain
7. Adding Monitoring Metrics
Example: Track acks
8. Implementing Tracing
Example: Inject traceparent
9. Adding Security Checks
Example: Redact in onSend
10. Transforming Messages
Example: Enrich value

34

Using Admin API

1. Creating Admin Client
Example: Create admin client
2. Creating Topics
Example: Create topic
3. Listing Topics
Example: List topics
4. Describing Topics
Example: Describe topic
5. Deleting Topics
Example: Delete topic
6. Creating Partitions
Example: Add partitions
7. Altering Configs
Example: Alter topic config
8. Describing Configs
Example: Describe config
9. Describing Consumer Groups
Example: Describe group
10. Listing Consumer Groups
Example: List groups
11. Deleting Consumer Groups
Example: Delete group
12. Deleting Records
Example: Delete records

35

Using Console Tools

1. Producing Messages
Example: Console produce
2. Consuming Messages
Example: Console consume
3. Reading from Beginning
Example: From beginning
4. Specifying Consumer Group
Example: With group
5. Setting Message Format
Example: Print key and value
6. Using Key Deserializer
Example: Long key deserializer
7. Setting Partition
Example: Specific partition
8. Reading Specific Offset
Example: Read from offset 500
9. Displaying Message Timestamp
Example: Show metadata
10. Using Timeout
Example: Timeout

36

Using Kafka CLI Tools

1. Managing Topics
Example: kafka-topics
2. Working with Consumer Groups
Example: kafka-consumer-groups
3. Managing ACLs
Example: kafka-acls
4. Managing Configs
Example: kafka-configs
5. Reassigning Partitions
Example: Reassign partitions
6. Preferred Leader Election
Example: Trigger election
7. Verifying Consumer Group
Example: Check lag
8. Verifying Producer
Example: Producer perf test
9. Dumping Log Segments
Example: Dump log
10. Mirroring Data
Example: Run MirrorMaker 2
11. Running Performance Tests
Example: Consumer perf test

37

Implementing Security

1. Understanding Kafka Security Model
Example: Security protocol
2. Enabling SSL Encryption
Example: Enable SSL
3. Configuring SSL Certificates
Example: Certificate config
4. Setting Keystore Password
Example: Externalized password
5. Configuring SSL Endpoint Identification
Example: Hostname verification
6. Enabling SASL Authentication
Example: Enable SASL_SSL
7. Configuring SASL/PLAIN
Example: SASL/PLAIN JAAS
8. Using SASL/SCRAM
Example: Create SCRAM user
9. Configuring SASL/GSSAPI
Example: GSSAPI JAAS
10. Setting Up SASL/OAUTHBEARER
Example: OAuth config
11. Enabling Inter-Broker Security
Example: Inter-broker SSL

38

Managing Access Control

1. Understanding ACL Model
Example: ACL anatomy
2. Enabling Authorization
Example: Enable authorizer
3. Creating ACLs
Example: Grant produce
4. Listing ACLs
Example: List ACLs for topic
5. Removing ACLs
Example: Remove ACL
6. Setting Principal Format
Example: Principal mapping rule
7. Configuring Super Users
Example: Super users
8. Understanding ACL Operations
Example: Consumer needs Read on group
9. Setting Resource Patterns
Example: Prefixed ACL
10. Using Wildcard ACLs
Example: All-topics read
11. Configuring ACL Binding
Example: ACL via Admin API

39

Configuring Quotas

1. Understanding Quotas Concept
Example: Quota throttling
2. Setting Producer Quotas
Example: Producer quota
3. Configuring Consumer Quotas
Example: Consumer quota
4. Setting Request Quotas
Example: Request quota
5. Configuring Client-Level Quotas
Example: Client.id quota
6. Setting User-Level Quotas
Example: User quota
7. Using Default Quotas
Example: Default user quota
8. Configuring Quota Window
Example: Quota window
9. Monitoring Quota Violations
Example: Throttle metric
10. Dynamic Quota Configuration
Example: Remove a quota

40

Implementing Dead Letter Queues

1. Understanding DLQ Pattern
Example: DLQ flow
2. Creating DLQ Topic
Example: Create DLQ
3. Routing Failed Messages
Example: Route to DLQ
4. Configuring DLQ in Streams
Example: Branch invalid records
5. Using DLQ in Connect
Example: Connect DLQ config
6. Adding Error Metadata
Example: Add error headers
7. Monitoring DLQ
Example: Watch DLQ lag
8. Reprocessing DLQ Messages
Example: Replay DLQ
9. Setting DLQ Retention
Example: 14-day DLQ retention
10. Implementing Circuit Breaker
Example: Pause on breaker open

41

Handling Errors and Exceptions

1. Understanding Producer Exceptions
Example: Callback handling
2. Handling Timeout Exceptions
Example: Catch timeout
3. Dealing with Serialization Errors
Example: Guard serialization
4. Handling Authorization Failures
Example: Handle auth failure
5. Managing Retriable Exceptions
Example: Safe retries
6. Handling Non-Retriable Errors
Example: Branch on retriability
7. Understanding Consumer Exceptions
Example: Wakeup shutdown
8. Handling Offset Out of Range
Example: Reset policy
9. Dealing with Rebalance Exceptions
Example: Commit before revoke
10. Configuring Error Handlers
Example: Streams uncaught handler

42

Optimizing Producer Performance

1. Tuning Batch Size
Example: Larger batch
2. Configuring Linger Time
Example: Linger for batching
3. Optimizing Compression
Example: Enable compression
4. Setting Buffer Memory
Example: Buffer memory
5. Configuring Max In-Flight Requests
Example: In-flight requests
6. Using Idempotent Producer
Example: Idempotence
7. Optimizing Serialization
Example: Compact serializer
8. Tuning Request Timeout
Example: Request timeout
9. Configuring Retries
Example: Durable producer
10. Monitoring Producer Metrics
Example: Producer JMX

43

Optimizing Consumer Performance

1. Tuning Fetch Size
Example: Larger fetch
2. Configuring Fetch Wait Time
Example: Fetch wait
3. Setting Max Poll Records
Example: Poll records
4. Optimizing Partition Assignment
Example: Assignment strategy
5. Using Parallel Processing
Example: Worker pool
6. Configuring Prefetching
Example: Receive buffer
7. Optimizing Deserialization
Example: Cache size for registry
8. Tuning Commit Frequency
Example: Commit interval
9. Using Asynchronous Commits
Example: Async + sync hybrid
10. Monitoring Consumer Metrics
Example: Consumer lag JMX

44

Optimizing Broker Performance

1. Tuning Number of Partitions
Example: Partition count
2. Optimizing Replica Fetcher Threads
Example: Replica fetchers
3. Configuring Network Threads
Example: Network threads
4. Tuning I/O Threads
Example: IO threads
5. Optimizing Log Segment Size
Example: Segment size
6. Configuring Zero-Copy Transfer
Example: Zero-copy note
7. Tuning Flush Settings
Example: Default flush (OS-managed)
8. Optimizing Page Cache
Example: JVM heap sizing
9. Setting Socket Buffer Sizes
Example: Socket buffers
10. Monitoring Broker Metrics
Example: Broker JMX

45

Optimizing Consumer Rebalancing

1. Understanding Rebalance Protocol
Example: Cooperative assignor
2. Configuring Session Timeout
Example: Session timeout
3. Setting Heartbeat Interval
Example: Heartbeat interval
4. Configuring Max Poll Interval
Example: Max poll interval
5. Using Cooperative Rebalancing
Example: Enable cooperative
6. Implementing Static Membership
Example: Static membership
7. Minimizing Rebalance Overhead
Example: Graceful leave
8. Using Incremental Rebalancing
Example: Partitions lost callback
9. Handling Rebalance Listeners
Example: Subscribe with listener
10. Monitoring Rebalance Latency
Example: Rebalance JMX

46

Monitoring Broker Metrics

1. Understanding JMX Metrics
Example: Enable JMX
2. Monitoring Bytes In/Out Rate
Example: Bytes in
3. Tracking Message Rate
Example: Messages in
4. Monitoring Request Metrics
Example: Request latency
5. Checking Under-Replicated Partitions
Example: URP metric
6. Monitoring ISR Shrink/Expand
Example: ISR shrink
7. Tracking Leader Election Rate
Example: Unclean elections
8. Monitoring Log Flush Latency
Example: Flush latency
9. Checking Network Processor Idle
Example: Network idle
10. Monitoring Request Queue Size
Example: Request queue

47

Monitoring Producer Metrics

1. Tracking Record Send Rate
Example: Send rate
2. Monitoring Batch Size
Example: Batch size avg
3. Checking Compression Ratio
Example: Compression ratio
4. Monitoring Record Queue Time
Example: Queue time
5. Tracking Request Latency
Example: Request latency
6. Monitoring Buffer Available Bytes
Example: Buffer available
7. Checking Waiting Threads
Example: Waiting threads
8. Tracking Record Error Rate
Example: Error rate
9. Monitoring Record Retry Rate
Example: Retry rate
10. Checking Metadata Age
Example: Metadata age

48

Monitoring Consumer Metrics

1. Tracking Fetch Rate
Example: Fetch rate
2. Monitoring Records Consumed
Example: Records consumed
3. Checking Consumer Lag
Example: Lag max
4. Monitoring Fetch Latency
Example: Fetch latency
5. Tracking Commit Rate
Example: Commit rate
6. Monitoring Join Rate
Example: Join rate
7. Checking Sync Rate
Example: Sync rate
8. Tracking Heartbeat Response Time
Example: Heartbeat time
9. Monitoring Last Poll Time
Example: Poll interval
10. Checking Assigned Partitions
Example: Assigned partitions

49

Managing Cluster Operations

1. Adding Brokers to Cluster
Example: New broker config
2. Removing Brokers from Cluster
Example: Reassign off broker 4
3. Expanding Cluster
Example: Throttled reassignment
4. Balancing Partition Load
Example: Generate balance plan
5. Rolling Restart
Example: Wait for URP=0
6. Upgrading Kafka Version
Example: Pin protocol version
7. Migrating to KRaft Mode
Example: Migration flag
8. Handling Broker Failures
Example: Check leadership after failure
9. Managing Disk Space
Example: Shrink retention
10. Configuring Rack Awareness
Example: Set broker rack

50

Using MirrorMaker for Replication

1. Understanding MirrorMaker Architecture
Example: Run MM2
2. Configuring MirrorMaker 2.0
Example: mm2.properties
3. Setting Source Cluster
Example: Source security
4. Setting Target Cluster
Example: Target RF
5. Configuring Topic Replication
Example: Topic selection
6. Setting Replication Flow
Example: Bidirectional flow
7. Configuring Consumer Groups Sync
Example: Group offset sync
8. Setting ACLs Replication
Example: ACL sync
9. Configuring Heartbeat Connectors
Example: Enable heartbeats
10. Monitoring Replication Lag
Example: MM2 latency metric

51

Implementing Multi-Datacenter Replication

1. Understanding Multi-DC Patterns
Example: Stretch cluster racks
2. Using Cluster Linking
Example: Create cluster link
3. Configuring Geo-Replication
Example: WAN compression
4. Setting Up Disaster Recovery
Example: Translate offsets on DR
5. Implementing Failover Strategy
Example: Seek on failover
6. Configuring Conflict Resolution
Example: Region-scoped key
7. Monitoring Cross-Cluster Lag
Example: Lag from heartbeats
8. Using Replicator
Example: Replicator connector
9. Configuring Network Latency Handling
Example: WAN-tuned producer
10. Testing Failover Procedures
Example: Verify DR consumer position

52

Deploying Kafka on Cloud Platforms

1. Using Amazon MSK
Example: MSK IAM client
2. Deploying on Confluent Cloud
Example: Confluent Cloud client
3. Using Azure Event Hubs
Example: Event Hubs client
4. Deploying on Google Cloud
Example: Strimzi Kafka CR
5. Configuring Cloud-Specific Settings
Example: Private bootstrap
6. Managing Cloud Kafka Security
Example: Managed Service for Kafka auth (GCP)
7. Scaling Cloud Kafka Clusters
Example: MSK scale CLI
8. Monitoring Cloud Deployments
Example: MSK open monitoring scrape
9. Optimizing Cloud Costs
Example: Enable tiered storage
10. Implementing Hybrid Cloud Setup
Example: On-prem to cloud mirror

53

Capacity Planning and Sizing

1. Estimating Message Throughput
Example: Throughput math
2. Calculating Partition Count
Example: Partition formula
3. Sizing Broker Hardware
Example: Heap vs page cache
4. Estimating Storage Requirements
Example: Storage estimate
5. Planning Network Bandwidth
Example: NIC sizing
6. Determining Replication Factor
Example: Durable topic
7. Sizing Consumer Groups
Example: Group sizing
8. Planning for Peak Load
Example: Peak headroom plan
9. Configuring Retention Policies
Example: 7-day retention
10. Monitoring Resource Utilization
Example: Disk usage check

54

Testing Kafka Applications

1. Using EmbeddedKafka
Example: EmbeddedKafka
2. Creating Test Producer
Example: Test producer
3. Creating Test Consumer
Example: Test consumer
4. Testing Streams Topology
Example: TopologyTestDriver
5. Mocking External Dependencies
Example: MockProducer
6. Testing Serialization
Example: Serde round-trip
7. Testing Error Scenarios
Example: Inject send error
8. Integration Testing
Example: KafkaContainer
9. Performance Testing
Example: Producer perf test
10. Load Testing
Example: Throttled soak test

55

Debugging and Troubleshooting

1. Enabling Debug Logging
Example: Dynamic log level
2. Using Broker Logs
Example: Tail server log
3. Checking Controller Logs
Example: Controller log grep
4. Analyzing State Change Logs
Example: State change for topic
5. Using Kafka Dump Log Tool
Example: Dump segment
6. Checking Topic Metadata
Example: Describe topic
7. Analyzing Consumer Lag
Example: Describe group lag
8. Using JMX Console
Example: Enable JMX
9. Tracing Request Path
Example: Request log timing fields
10. Debugging Rebalance Issues
Example: Stabilize rebalances

56

Implementing Backup and Recovery

1. Understanding Backup Strategies
Example: Why RF isn't backup
2. Backing Up Topic Data
Example: S3 sink backup
3. Backing Up Consumer Offsets
Example: Export offsets
4. Backing Up Schemas
Example: Export all schemas
5. Backing Up Configuration
Example: Backup topic configs
6. Implementing Point-in-Time Recovery
Example: Reset group to timestamp
7. Restoring from Backup
Example: S3 source restore
8. Recovering Consumer Groups
Example: Restore offsets from file
9. Testing Recovery Procedures
Example: Verify restored count
10. Automating Backup Process
Example: Cron config backup

Visit our Github profile Visit our Linkedin profile Visit our Twitter profile

© 2026 MapagamLast updated: Jul 29, 2026