Managing Topics
1. Creating Topics
| Flag |
Purpose |
Example |
| --create |
Create a new topic |
Required action |
| --partitions |
Number of partitions |
6 |
| --replication-factor |
Copies per partition |
3 |
| --config |
Per-topic override |
retention.ms=86400000 |
Example: Create with configs
kafka-topics.sh --create --topic payments \
--partitions 12 --replication-factor 3 \
--config retention.ms=604800000 --config min.insync.replicas=2 \
--bootstrap-server localhost:9092
2. Listing Topics
| Flag |
Purpose |
Notes |
| --list |
Show all topic names |
One per line |
| --exclude-internal |
Hide __ topics |
Skip internal |
| --topic |
Filter by name pattern |
Exact match |
Example: List topics
kafka-topics.sh --list --bootstrap-server localhost:9092 --exclude-internal
3. Describing Topics
| Field |
Meaning |
Detail |
| Leader |
Broker serving the partition |
Per partition |
| Replicas |
All brokers holding a copy |
Order = preference |
| Isr |
In-sync replica set |
Election candidates |
Example: Describe a topic
kafka-topics.sh --describe --topic payments \
--bootstrap-server localhost:9092
4. Setting Partition Count
| Aspect |
Rule |
Detail |
| Increase Only |
Partitions can be added, never removed |
One-way operation |
| Key Ordering |
Adding partitions changes key→partition mapping |
Breaks co-location |
| Parallelism |
More partitions = more consumer scaling |
Match to throughput |
Example: Add partitions
kafka-topics.sh --alter --topic payments --partitions 24 \
--bootstrap-server localhost:9092
5. Setting Replication Factor
| Aspect |
Description |
Detail |
| At Creation |
Set via --replication-factor |
RF ≤ broker count |
| After Creation |
Requires reassignment JSON |
No direct flag |
| Durability |
RF=3 standard for production |
Tolerates 2 losses |
Example: Increase RF via reassignment
{"version":1,"partitions":[
{"topic":"payments","partition":0,"replicas":[1,2,3]}
]}
6. Altering Topic Configuration
| Action |
Command |
Notes |
| Set |
--add-config key=value |
Override default |
| Remove |
--delete-config key |
Revert to default |
| Describe |
--describe --all |
Show effective config |
Example: Alter retention
kafka-configs.sh --alter --entity-type topics --entity-name payments \
--add-config retention.ms=259200000 \
--bootstrap-server localhost:9092
7. Deleting Topics
| Aspect |
Description |
Detail |
| --delete |
Marks topic for deletion |
Async cleanup |
| delete.topic.enable |
Must be true on broker |
Default true |
| Irreversible |
All data and offsets removed |
No recovery |
Warning: Topic deletion is permanent and removes all partition data. Confirm before running in production.
Example: Delete a topic
kafka-topics.sh --delete --topic obsolete-topic \
--bootstrap-server localhost:9092
8. Configuring Topic Retention
| Config |
Description |
Default |
| retention.ms |
Time before deletion |
604800000 (7d) |
| retention.bytes |
Max partition size |
-1 (unlimited) |
| -1 value |
Disables that limit |
Infinite retention |
Example: Time + size retention
kafka-configs.sh --alter --entity-type topics --entity-name logs \
--add-config retention.ms=86400000,retention.bytes=1073741824 \
--bootstrap-server localhost:9092
9. Setting Cleanup Policy
| Policy |
Behavior |
Use Case |
| delete |
Remove old segments by age/size |
Event streams |
| compact |
Keep latest value per key |
Changelogs, state |
| compact,delete |
Compact then enforce retention |
Bounded changelog |
Example: Enable compaction
kafka-configs.sh --alter --entity-type topics --entity-name user-profiles \
--add-config cleanup.policy=compact \
--bootstrap-server localhost:9092
10. Configuring Segment Settings
| Config |
Description |
Default |
| segment.bytes |
Max segment file size |
1073741824 (1GB) |
| segment.ms |
Force roll after time |
604800000 (7d) |
| segment.index.bytes |
Max index file size |
10485760 |
Example: Smaller segments for faster compaction
kafka-configs.sh --alter --entity-type topics --entity-name user-profiles \
--add-config segment.bytes=104857600,segment.ms=3600000 \
--bootstrap-server localhost:9092