Implementing Message Queue Logic
1. Designing Message Producer
Example: Kafka producer
@Service @RequiredArgsConstructor
public class OrderProducer {
private final KafkaTemplate<String, OrderPlaced> kafka;
public void publish(OrderPlaced e) {
kafka.send("orders", e.orderId().toString(), e);
}
}
2. Designing Message Consumer
Example: @KafkaListener
@KafkaListener(topics = "orders", groupId = "billing")
public void onOrder(OrderPlaced e, Acknowledgment ack) {
billing.charge(e);
ack.acknowledge(); // manual commit after success
}
3. Implementing Message Acknowledgment
| Mode | Detail |
|---|---|
| Auto | Fast but at-most-once risk |
| Manual after success | At-least-once |
| Batch ack | Higher throughput |
| Negative ack (nack) | Trigger redelivery / DLQ |
4. Implementing Dead Letter Queues
| Aspect | Detail |
|---|---|
| When to send | After max retries |
| Include | Original message + error + attempts |
| Alerting | On DLQ depth growth |
| Replay tooling | Move from DLQ back to main |
5. Implementing Message Retry Logic
Example: Spring Retry on consumer
@RetryableTopic(
attempts = "5",
backoff = @Backoff(delay = 1000, multiplier = 2.0),
dltStrategy = DltStrategy.FAIL_ON_ERROR)
@KafkaListener(topics = "orders")
public void onOrder(OrderPlaced e) { billing.charge(e); }
6. Designing Message Format
| Format | Trade-off |
|---|---|
| JSON | Human-readable, larger |
| Avro | Schema registry, compact |
| Protobuf | Fast, compact, typed |
| MessagePack | Binary JSON-like |
7. Implementing Message Routing
| Pattern | Detail |
|---|---|
| Topic-based | Kafka topics |
| Direct exchange | RabbitMQ routing key match |
| Fanout | Broadcast to all queues |
| Header-based | Match on metadata |
| Content-based | Inspect payload |
8. Implementing Topic Subscription
| Pattern | Detail |
|---|---|
| Single consumer group | Load-balanced delivery |
| Multiple groups | Each gets full stream |
| Partition assignment | Range / round-robin / sticky |
| Wildcards | Topic pattern matching |
9. Handling Message Ordering
| System | How |
|---|---|
| Kafka | Per-partition order; partition by key |
| RabbitMQ | Single queue + single consumer |
| SQS FIFO | MessageGroupId for ordering |
| Trade-off | Ordering reduces parallelism |
10. Implementing Message Persistence
| System | Detail |
|---|---|
| Kafka | Durable log; configurable retention |
| RabbitMQ | Durable queues + persistent messages |
| Replication factor | ≥ 3 for production |
| Acks | Producer acks=all for safety |
11. Implementing Producer Retry Logic
| Setting | Detail |
|---|---|
| retries | Max attempts on transient errors |
| retry.backoff.ms | Spacing |
| enable.idempotence | Prevents duplicates on retry (Kafka) |
| max.in.flight | Set to 1 for strict order with retries |
12. Implementing Consumer Group Management
| Concept | Detail |
|---|---|
| Group ID | Identifies logical consumer |
| Offset commit | Per group per partition |
| Rebalance | On member join/leave |
| Static membership | Reduce rebalance churn |
| Lag monitoring | Critical health metric |