Managing Consumer Groups
1. Understanding Group Coordinator
| Concept |
Description |
Detail |
| Coordinator |
Broker managing a group's membership & offsets |
One per group |
| Selection |
Hash of group.id → __consumer_offsets partition |
Deterministic |
| Leader Member |
One consumer computes assignment |
Group leader |
Example: Find coordinator
kafka-consumer-groups.sh --describe --group billing \
--members --bootstrap-server localhost:9092
2. Understanding Rebalancing
| Trigger |
Cause |
Effect |
| Member Join/Leave |
Scaling or crash |
Reassign partitions |
| Topic Change |
Partition added |
New assignment |
| Stop-the-world |
Eager protocol pauses all |
Use cooperative |
Example: Rebalance listener
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
public void onPartitionsRevoked(Collection<TopicPartition> p) { commit(); }
public void onPartitionsAssigned(Collection<TopicPartition> p) { /* init */ }
});
3. Listing Consumer Groups
| Flag |
Purpose |
Detail |
| --list |
Show all group IDs |
One per line |
| --state |
Include group state |
Stable/Empty |
| --all-groups |
Apply to every group |
Bulk ops |
Example: List groups
kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
4. Describing Consumer Groups
| Column |
Meaning |
Detail |
| CURRENT-OFFSET |
Last committed offset |
Per partition |
| LOG-END-OFFSET |
Latest produced offset |
Tip of log |
| LAG |
END minus CURRENT |
Backlog |
Example: Describe a group
kafka-consumer-groups.sh --describe --group billing \
--bootstrap-server localhost:9092
5. Viewing Consumer Lag
| Metric |
Description |
Detail |
| Partition Lag |
END - committed per partition |
Granular |
| Total Lag |
Sum across partitions |
Overall health |
| records-lag-max |
JMX consumer metric |
Live monitoring |
Example: Check lag
kafka-consumer-groups.sh --describe --group billing \
--bootstrap-server localhost:9092 | awk '{print $1,$3,$6}'
6. Resetting Offsets
| Option |
Target |
Detail |
| --to-earliest |
Oldest offset |
Full replay |
| --to-latest |
Newest offset |
Skip backlog |
| --shift-by |
Relative move |
+/- N |
| --dry-run |
Preview only |
Then --execute |
Warning: Offset reset requires the group to be inactive (no live members) or it will fail.
Example: Reset to earliest
kafka-consumer-groups.sh --reset-offsets --group billing \
--topic orders --to-earliest --execute \
--bootstrap-server localhost:9092
7. Deleting Consumer Groups
| Aspect |
Description |
Constraint |
| --delete |
Remove group + offsets |
Must be empty |
| Active Group |
Cannot delete with members |
Stop consumers |
| Effect |
Re-consume per offset reset |
State lost |
Example: Delete a group
kafka-consumer-groups.sh --delete --group old-group \
--bootstrap-server localhost:9092
8. Configuring Session Timeout
| Config |
Scope |
Default |
| session.timeout.ms |
Consumer-side liveness |
45000 |
group.min.session.
timeout.ms |
Broker lower bound |
6000 |
group.max.session.
timeout.ms |
Broker upper bound |
1800000 |
Example: Broker bounds
group.min.session.timeout.ms=6000
group.max.session.timeout.ms=300000
9. Setting Heartbeat Interval
| Config |
Description |
Detail |
| heartbeat.interval.ms |
Frequency to coordinator |
3000 |
| Ratio |
1/3 of session timeout |
Guideline |
| Missed |
Triggers eviction after session |
Rebalance |
Example: Align heartbeat
props.put("session.timeout.ms", 30000);
props.put("heartbeat.interval.ms", 10000);
10. Configuring Max Poll Interval
| Config |
Description |
Detail |
| max.poll.interval.ms |
Processing budget per batch |
300000 |
| Symptom |
Repeated rebalances if exceeded |
Tune work |
| Mitigation |
Lower max.poll.records |
Or offload |
Example: Prevent eviction during heavy work
props.put("max.poll.interval.ms", 900000);
props.put("max.poll.records", 50);