Understanding Message Ordering
1. Understanding Ordering Guarantees
| Scope | Guarantee | Detail |
|---|---|---|
| Partition | Strict offset order | Guaranteed |
| Topic | No cross-partition order | Not guaranteed |
| Key | Same key → same partition → ordered | Practical ordering |
Example: Per-key ordering
// All events for account 7 are ordered (same partition)
producer.send(new ProducerRecord<>("ledger", "acct-7", entryJson));
2. Preserving Order with Single Partition
| Aspect | Description | Tradeoff |
|---|---|---|
| 1 Partition | Total order across topic | No parallelism |
| Throughput | Single consumer max | Limited |
| Use Case | Strict global sequence | Low volume |
Example: Single-partition topic
kafka-topics.sh --create --topic audit-log --partitions 1 \
--replication-factor 3 --bootstrap-server localhost:9092
3. Using Message Keys for Ordering
| Aspect | Description | Detail |
|---|---|---|
| Key Choice | Entity ID groups events | Per-entity order |
| Parallelism | Different keys spread out | Scalable |
| Repartition Risk | Adding partitions breaks it | Plan upfront |
4. Configuring Max In-Flight Requests
| Setting | Effect | Detail |
|---|---|---|
| =1 | Strict order without idempotence | Slower |
| ≤5 + idempotence | Order preserved, faster | Recommended |
| >5 no idempotence | Reorder on retry | Risky |
Example: Safe in-flight + idempotence
props.put("enable.idempotence", "true");
props.put("max.in.flight.requests.per.connection", 5);
5. Enabling Idempotent Producer
| Aspect | Description | Detail |
|---|---|---|
| Idempotence | No dup + no reorder on retry | Per partition |
| Sequence Numbers | Broker tracks per PID | Detects gaps |
| Default | Enabled by default | 3.0+ |
Example: Idempotent producer
props.put("enable.idempotence", "true"); // ordering preserved on retries
6. Understanding Ordering Across Partitions
| Aspect | Description | Detail |
|---|---|---|
| No Global Order | Partitions consumed independently | By design |
| Merge Need | App must reorder if required | Buffer + sort |
| Timestamps | Use event time to reconcile | Approximate |
Example: Reorder by timestamp buffer
PriorityQueue<ConsumerRecord<String,String>> pq =
new PriorityQueue<>(Comparator.comparingLong(ConsumerRecord::timestamp));
records.forEach(pq::add);
7. Implementing Total Ordering
| Approach | Description | Tradeoff |
|---|---|---|
| Single Partition | Simplest total order | No scaling |
| Sequence Field | Embed global seq, reorder downstream | Complex |
| Merge Consumer | Collect + sort all partitions | Latency |
Example: Embed sequence number
long seq = sequencer.next();
r.headers().add("seq", Long.toString(seq).getBytes());
8. Handling Out-of-Order Messages
| Strategy | Description | Detail |
|---|---|---|
| Grace Period | Streams waits for late data | Windowed |
| Buffering | Hold and sort window | Memory cost |
| Idempotent Apply | Tolerate reordering | CRDT-like |
Example: Streams grace period
TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1));
9. Using Timestamps for Ordering
| Aspect | Description | Detail |
|---|---|---|
| Event Time | Order by CreateTime | Logical order |
| Watermark | Bound lateness | Streams concept |
| Sort Window | Reorder within window | Approximate |
Example: Custom timestamp extractor
public long extract(ConsumerRecord<Object,Object> r, long prev) {
return ((Event) r.value()).getEventTimeMs();
}
10. Configuring Retries for Ordering
| Setting | Effect | Detail |
|---|---|---|
| retries > 0 | Safe with idempotence | No reorder |
| Without Idempotence | Retry can reorder batches | Use inflight=1 |
| retry.backoff.ms | Spacing between retries | 100 default |