Working with Message Queues

1. Understanding Message Queue Patterns

Pattern	Description
Point-to-point	One producer, one consumer per msg (queue)
Pub/Sub	One producer, many subscribers (topic/fanout)
Competing consumers	Many workers share one queue
Request-reply	Reply queue + correlation id
Routing	Topic/header-based routing (RabbitMQ exchanges)

2. Implementing Producer-Consumer Pattern

Example: Kafka producer/consumer

// Producer
KafkaProducer<String, String> p = new KafkaProducer<>(props);
p.send(new ProducerRecord<>("orders", order.id, json));

// Consumer
KafkaConsumer<String, String> c = new KafkaConsumer<>(props);
c.subscribe(List.of("orders"));
while (true) {
    ConsumerRecords<String, String> recs = c.poll(Duration.ofMillis(500));
    for (var r : recs) handle(r);
    c.commitSync();
}

3. Implementing Message Acknowledgments

Mode	Semantics
Auto-ack	Acked on receive; risk of loss
Manual ack after process	At-least-once
Negative ack (nack)	Reject + requeue / DLQ
Cumulative ack	Single ack covers offset range (Kafka)

4. Implementing Exactly-Once Semantics

Mechanism	System
Idempotent producer + transactions	Kafka EOS (read-process-write)
Dedup at consumer (idempotency key)	App layer
Two-phase commit with sink	Flink, Beam
Outbox pattern	DB tx writes msg + state

Warning: Pure exactly-once across boundaries is impossible (Two Generals); achieve "effectively once" via idempotency + dedup.

5. Handling Message Ordering Guarantees

Guarantee	System
Per-partition FIFO	Kafka, Kinesis
Per-queue FIFO	SQS FIFO, Azure Service Bus sessions
Group ordering	RabbitMQ single consumer; partition by key
Global ordering	Single partition (low throughput)

6. Implementing Dead Letter Queues

Trigger	Action
Max retries exceeded	Move to DLQ
TTL expired	Move to DLQ
Queue length limit	Drop or DLQ
Operator action	Inspect, replay, or discard
Alerts	Monitor DLQ depth as SLI

7. Implementing Message Expiration (TTL)

Scope	Mechanism
Per-message TTL	Header (RabbitMQ `expiration`)
Per-queue TTL	Default for all in queue
Topic retention	Kafka `retention.ms`
SQS	Visibility timeout + 14-day max

8. Implementing Priority Queues

Approach	Detail
Priority field	RabbitMQ `x-max-priority`
Multiple queues per level	Workers poll high → low
Weighted fair queue	Round-robin with weights

9. Handling Poison Messages

Step	Action
Detect	Counter on retries; consistent failure
Quarantine	Move to DLQ after threshold
Alert	Engineer investigates payload
Replay	After fix, re-enqueue from DLQ

10. Implementing Message Batching

Knob	Effect
Batch size	Bigger = better throughput, higher latency
Linger time	Wait to fill batch (Kafka `linger.ms`)
Compression	gzip, snappy, lz4, zstd — applied per batch
Trade-off	Throughput ↑, end-to-end latency ↑

11. Understanding Message Broker Architectures

Broker	Architecture	Best For
Kafka	Distributed log, partitioned	High throughput, replay
RabbitMQ	Smart broker, routing	Complex routing, low-mid scale
NATS / JetStream	Lightweight pub/sub	Microservices, IoT
Pulsar	Segmented log + tiered storage	Multi-tenant, geo-replication
SQS / Pub/Sub / EventBridge	Managed, serverless	Cloud-native, no ops
Redis Streams	In-memory log	Low-latency, smaller scale

Related Cheat Sheets

REST API

Working with Message Queues

1. Understanding Message Queue Patterns

2. Implementing Producer-Consumer Pattern

Example: Kafka producer/consumer

3. Implementing Message Acknowledgments

4. Implementing Exactly-Once Semantics

5. Handling Message Ordering Guarantees

6. Implementing Dead Letter Queues

7. Implementing Message Expiration (TTL)

8. Implementing Priority Queues

9. Handling Poison Messages

10. Implementing Message Batching

11. Understanding Message Broker Architectures

Related Cheat Sheets

Structuring Request Payloads

Using Web Streams API

Optimizing Performance

Implementing Secure Authentication Flows