Understanding Message Ordering

1. Understanding Ordering Guarantees

Scope Guarantee Detail
Partition Strict offset order Guaranteed
Topic No cross-partition order Not guaranteed
Key Same key → same partition → ordered Practical ordering

Example: Per-key ordering

// All events for account 7 are ordered (same partition)
producer.send(new ProducerRecord<>("ledger", "acct-7", entryJson));

2. Preserving Order with Single Partition

Aspect Description Tradeoff
1 Partition Total order across topic No parallelism
Throughput Single consumer max Limited
Use Case Strict global sequence Low volume

Example: Single-partition topic

kafka-topics.sh --create --topic audit-log --partitions 1 \
  --replication-factor 3 --bootstrap-server localhost:9092

3. Using Message Keys for Ordering

Aspect Description Detail
Key Choice Entity ID groups events Per-entity order
Parallelism Different keys spread out Scalable
Repartition Risk Adding partitions breaks it Plan upfront

Example: Key-based ordering

producer.send(new ProducerRecord<>("orders", customerId, orderEvent));

4. Configuring Max In-Flight Requests

Setting Effect Detail
=1 Strict order without idempotence Slower
≤5 + idempotence Order preserved, faster Recommended
>5 no idempotence Reorder on retry Risky

Example: Safe in-flight + idempotence

props.put("enable.idempotence", "true");
props.put("max.in.flight.requests.per.connection", 5);

5. Enabling Idempotent Producer

Aspect Description Detail
Idempotence No dup + no reorder on retry Per partition
Sequence Numbers Broker tracks per PID Detects gaps
Default Enabled by default 3.0+

Example: Idempotent producer

props.put("enable.idempotence", "true"); // ordering preserved on retries

6. Understanding Ordering Across Partitions

Aspect Description Detail
No Global Order Partitions consumed independently By design
Merge Need App must reorder if required Buffer + sort
Timestamps Use event time to reconcile Approximate

Example: Reorder by timestamp buffer

PriorityQueue<ConsumerRecord<String,String>> pq =
    new PriorityQueue<>(Comparator.comparingLong(ConsumerRecord::timestamp));
records.forEach(pq::add);

7. Implementing Total Ordering

Approach Description Tradeoff
Single Partition Simplest total order No scaling
Sequence Field Embed global seq, reorder downstream Complex
Merge Consumer Collect + sort all partitions Latency

Example: Embed sequence number

long seq = sequencer.next();
r.headers().add("seq", Long.toString(seq).getBytes());

8. Handling Out-of-Order Messages

Strategy Description Detail
Grace Period Streams waits for late data Windowed
Buffering Hold and sort window Memory cost
Idempotent Apply Tolerate reordering CRDT-like

Example: Streams grace period

TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1));

9. Using Timestamps for Ordering

Aspect Description Detail
Event Time Order by CreateTime Logical order
Watermark Bound lateness Streams concept
Sort Window Reorder within window Approximate

Example: Custom timestamp extractor

public long extract(ConsumerRecord<Object,Object> r, long prev) {
    return ((Event) r.value()).getEventTimeMs();
}

10. Configuring Retries for Ordering

Setting Effect Detail
retries > 0 Safe with idempotence No reorder
Without Idempotence Retry can reorder batches Use inflight=1
retry.backoff.ms Spacing between retries 100 default

Example: Order-preserving retries

props.put("enable.idempotence", "true");
props.put("retries", Integer.MAX_VALUE);
props.put("max.in.flight.requests.per.connection", 5);