Designing Message Queue Architecture
1. Understanding Message Queue Patterns
| Pattern | Use Case |
|---|---|
| Work queue | 1 of N consumers gets each msg |
| Pub/sub | All subscribers get each msg |
| Routing | Topic/header-based filtering |
| RPC | Reply queue + correlation_id |
| Streams | Replayable log (Kafka) |
2. Designing Message Broker Architecture
| Broker | Strength |
|---|---|
| Kafka | High throughput, replayable log |
| RabbitMQ | Rich routing, AMQP, work queues |
| NATS / NATS JetStream | Low latency, simple |
| Pulsar | Tiered storage, geo-replication |
| SQS / SNS | Managed, simple |
| Redis Streams | Light, in-memory |
3. Designing Message Ordering and Delivery Guarantees
| Guarantee | Mechanism |
|---|---|
| FIFO | Partition by key (Kafka), FIFO queue (SQS) |
| At-most-once | Fire-and-forget; possible loss |
| At-least-once | ack required; possible duplicate |
| Exactly-once | Idempotent producer + transactions |
4. Designing Dead Letter Queue Strategy
| Setting | Recommendation |
|---|---|
| Trigger | After N failed attempts (3–5) |
| DLQ retention | 14 days+ for triage |
| Alerting | Page on DLQ depth > 0 |
| Replay tooling | Move back to main queue after fix |
5. Designing Message Idempotency
| Technique | Detail |
|---|---|
| Dedup table | (message_id, processed_at) |
| Conditional updates | Use UPSERT / CAS |
| Producer-assigned ID | Stable across retries |
| Bloom filter window | Memory-efficient recent-dup check |
6. Designing Message Schema Evolution
| Format | Evolution Rules |
|---|---|
| Avro | Default values for new fields |
| Protobuf | Don't reuse field nums; reserved |
| JSON Schema | Optional fields; ignore unknown |
| Schema registry | Enforce compatibility on publish |
7. Understanding At-Least-Once vs Exactly-Once Delivery
| Mode | Trade-off |
|---|---|
| At-least-once + idempotent | Practical "exactly once" outcome |
| Exactly-once (Kafka EOS) | Higher latency, transactional overhead |
| At-most-once | Best perf; OK for metrics, logs |
8. Designing Message Priority Queues
| Approach | Detail |
|---|---|
| Multiple queues | high/medium/low; consumer drains in order |
| Native priority | RabbitMQ x-max-priority |
| Weighted fair | Avoid starvation of low priority |
| Caveat | Strict priority can starve; cap budget per tier |
9. Designing Message TTL and Expiration
| Setting | Detail |
|---|---|
| Message TTL | Drop if older than X |
| Queue TTL | Auto-delete idle queue |
| Expired action | Drop, DLQ, or callback |
10. Designing Message Acknowledgment Strategy
| Mode | Detail |
|---|---|
| Auto-ack | Ack on receive; risk of loss |
| Manual after success | Safe; standard |
| Negative ack (NACK) | Requeue or DLQ |
| Visibility timeout (SQS) | Hide while processing; redeliver if not acked |
11. Designing Message Retry Strategy
| Mechanism | Detail |
|---|---|
| In-memory retry | Fast, but lost on crash |
| Delay queue | Visibility delay or scheduled topic |
| Exponential backoff queues | retry-1m → retry-5m → retry-30m → DLQ |
| Max attempts | 3–5 then DLQ |
12. Designing Queue Partitioning
| Concept | Detail |
|---|---|
| Partitions | Parallelism unit (Kafka) |
| Partition key | Hashed to choose partition |
| Consumer group | One consumer per partition (within group) |
| Rebalance | On consumer add/remove |
| Hot partition | Avoid skewed keys |