Managing Offsets

1. Understanding Offset Commits

Concept Description Detail
Commit Records position in __consumer_offsets Per partition
Committed Offset Next offset to read on restart position, not last read
Delivery Commit timing sets semantics At-least/at-most

Example: Commit after processing

for (var r : records) process(r);
consumer.commitSync(); // at-least-once

2. Enabling Auto Commit

Config Description Default
enable.auto.commit Periodic background commit true
Risk Commits before processing done Possible loss
Simplicity No manual commit code Convenience

Example: Enable auto commit

props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "5000");

3. Setting Auto Commit Interval

Config Description Default
auto.commit.interval.ms Commit frequency 5000
Trigger Committed during poll() Not exact timer
Tradeoff Lower = less reprocessing More overhead

Example: Frequent auto commit

props.put("auto.commit.interval.ms", "1000");

4. Committing Synchronously

Method Behavior Detail
commitSync() Blocks until broker confirms Retries internally
Reliability Throws on failure Easy to handle
Throughput Slower per batch Pauses loop

Example: Synchronous commit

try {
    consumer.commitSync();
} catch (CommitFailedException e) {
    log.warn("commit failed, will retry next loop", e);
}

5. Committing Asynchronously

Method Behavior Detail
commitAsync() Non-blocking, callback on result No internal retry
Throughput Higher, loop continues Fire-and-forget
Pattern Async in loop + sync on close Best of both

Example: Async with final sync

try {
    while (running) { /* poll, process */ consumer.commitAsync(); }
} finally {
    consumer.commitSync(); // ensure last offsets committed
    consumer.close();
}

6. Committing Specific Offsets

Aspect Description Detail
Map argument Commit chosen TopicPartition→offset Fine-grained
offset+1 Commit next offset to read Common mistake: off by one
Per-record Commit mid-batch Reduce reprocessing

Example: Commit explicit offsets

Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
offsets.put(new TopicPartition(r.topic(), r.partition()),
    new OffsetAndMetadata(r.offset() + 1));
consumer.commitSync(offsets);

7. Storing Offsets Externally

Aspect Description Detail
External Store Save offsets in DB with data Atomic with state
seek() on start Restore via rebalance listener Exactly-once sink
Disable Auto enable.auto.commit=false Required

Example: Restore offset from DB

public void onPartitionsAssigned(Collection<TopicPartition> parts) {
    for (TopicPartition tp : parts)
        consumer.seek(tp, offsetStore.read(tp)); // from external DB
}

8. Resetting to Earliest

Aspect Description Detail
seekToBeginning Programmatic earliest In-app
--to-earliest CLI group reset Offline
Effect Reprocess all retained data Full replay

Example: Seek to beginning

consumer.seekToBeginning(consumer.assignment());

9. Resetting to Latest

Aspect Description Detail
seekToEnd Programmatic latest Skip backlog
--to-latest CLI group reset Offline
Use Case Only new events matter Drop history

Example: Skip to latest

kafka-consumer-groups.sh --reset-offsets --group billing \
  --topic orders --to-latest --execute --bootstrap-server localhost:9092

10. Resetting to Specific Offset

Option Description Detail
--to-offset Exact offset value Per partition
--shift-by Relative ± N Replay window
--topic t:p Scope to partition Granular

Example: Reset to offset 1000

kafka-consumer-groups.sh --reset-offsets --group billing \
  --topic orders:0 --to-offset 1000 --execute \
  --bootstrap-server localhost:9092

11. Resetting by Timestamp

Option Description Detail
--to-datetime Reset to first offset ≥ time ISO-8601
offsetsForTimes API timestamp→offset lookup Programmatic
Use Case Replay from incident time Recovery

Example: Reset by datetime

kafka-consumer-groups.sh --reset-offsets --group billing \
  --topic orders --to-datetime 2026-06-15T00:00:00.000 \
  --execute --bootstrap-server localhost:9092

Related Cheat Sheets

Kafka

Installing and Configuring Kafka

Kafka

Understanding Message Ordering

Kafka

Configuring Quotas

Kafka

Using Kafka CLI Tools