Working with Message Brokers
1. Implementing Publish-Subscribe Pattern
| Element | Description |
|---|---|
| Publisher | Emits to topic; unaware of subscribers |
| Topic | Logical channel |
| Subscriber | One copy per subscriber group |
| Fan-out | 1 message → N consumers |
| Examples | Kafka topic, RabbitMQ fanout exchange, SNS |
2. Implementing Point-to-Point Pattern
| Element | Description |
|---|---|
| Sender | Posts to queue |
| Queue | Persistent FIFO buffer |
| Receiver | Single consumer wins each message (competing consumers) |
| Use case | Work distribution: jobs, tasks |
| Examples | SQS, RabbitMQ direct, Kafka w/ same group |
3. Using Message Headers
| Header | Use |
|---|---|
| message-id | Unique ID for dedup |
| correlation-id | Trace business workflow |
| content-type | application/json, application/avro |
| schema-id / schema-version | Schema registry reference |
| timestamp | Producer event time |
| x-retry-count | Retry tracking |
4. Implementing Message Routing
| Strategy | Detail |
|---|---|
| Direct (routing key) | Exact-match key → queue |
| Topic (wildcard) | order.*.eu patterns |
| Header-based | Match on header values |
| Content-based | Router inspects payload |
| Kafka partitioning | Hash of key → partition |
5. Handling Message Acknowledgment
| Mode | Semantics |
|---|---|
| Auto-ack | Ack on receive (at-most-once; lossy) |
| Manual ack | Ack after success (at-least-once) |
| Negative ack (nack) | Reject & requeue or DLQ |
| Kafka commit | Commit offsets after processing batch |
| SQS visibility timeout | Hidden until processed/timeout |
6. Implementing Dead Letter Queues
| Aspect | Detail |
|---|---|
| Trigger | Max delivery attempts exceeded |
| Storage | Separate queue / topic |
| Metadata | Failure reason, attempt count, original headers |
| Action | Manual replay, alert, archive |
| Tip | Alert on DLQ depth > 0 |
7. Using Message TTL
| Setting | Detail |
|---|---|
| Per-message TTL | Drop after N ms in queue |
| Per-queue TTL | Default for all messages |
| Kafka retention | Time or size based (retention.ms) |
| Use case | Stale order events, time-sensitive notifications |
| Expired | Drop or route to DLQ |
8. Implementing Message Ordering
| Mechanism | Detail |
|---|---|
| Single partition/queue | Strict FIFO; no parallelism |
| Partition key | Same key → same partition (Kafka) |
| FIFO queues (SQS) | Per MessageGroupId |
| RabbitMQ | Per-queue ordering with single consumer |
| Trade-off | Ordering ⊥ throughput |
9. Handling Poison Messages
| Step | Action |
|---|---|
| Detect | Repeated processing failures |
| Limit retries | Max delivery count (e.g. 5) |
| Quarantine | Move to DLQ with diagnostics |
| Alert | Page on-call if rate spikes |
| Replay | After fix, re-publish from DLQ |
10. Using Message Priority
| Approach | Detail |
|---|---|
| Priority queues | RabbitMQ x-max-priority |
| Multiple queues | High/medium/low; weighted poll |
| Kafka workaround | Separate topics per priority |
| Caveat | Risk of starving low-priority work |
11. Implementing Message Deduplication
| Technique | Detail |
|---|---|
| Producer dedup | Kafka idempotent producer (PID + seq) |
| SQS FIFO dedup | MessageDeduplicationId within 5 min window |
| Consumer dedup | Store processed message-id in Redis with TTL |
| Idempotent handler | Use natural keys; safe to re-run |
12. Working with RabbitMQ, Kafka, AWS SQS/SNS
| Feature | RabbitMQ | Kafka | SQS / SNS |
|---|---|---|---|
| Model | Queue + Exchange | Distributed log | Queue (SQS) / Pub-Sub (SNS) |
| Throughput | 10–100k msg/s | Millions msg/s | Unlimited (managed) |
| Replay | No (consumed = gone) | Yes (retain by time/size) | No |
| Ordering | Per-queue | Per-partition | FIFO queues |
| Best for | Routing, RPC, work queues | Streaming, event sourcing | Cloud-native decoupling |