Working with Event-Driven Patterns
1. Implementing Event Streaming
| Platform | Strength |
|---|---|
| Kafka | High throughput, log replay |
| Pulsar | Multi-tenant, geo-replication |
| NATS JetStream | Lightweight, simple |
| AWS Kinesis | Managed, shards |
| Redpanda | Kafka-compatible, no ZK |
2. Configuring Message Queue Integration
Example: Gateway → Kafka (HTTP-source)
routes:
- paths: ["/events"]
methods: [POST]
plugins:
- name: kafka-upstream
config:
bootstrap_servers: [kafka-1:9092, kafka-2:9092]
topic: api-events
authentication:
mechanism: SCRAM-SHA-512
user: gateway
password: ${KAFKA_PASS}
producer_async: true
producer_request_acks: 13. Using Pub/Sub Patterns
| Pattern | Detail |
|---|---|
| Topic-based | Subscribers per topic |
| Content-based | Filter by message attributes |
| Fanout | One msg → many subscribers |
| Queue (point-to-point) | One msg → one consumer |
4. Implementing Event Routing
| Route By | Example |
|---|---|
| Event type | order.created → orders topic |
| Tenant | Per-tenant partition |
| Priority | High → fast queue |
| Region | Geo affinity |
5. Setting Up Event Transformation
| Transform | Purpose |
|---|---|
| CloudEvents wrapping | Standardize metadata |
| Envelope adding | Add trace, source |
| Schema mapping | v1 → v2 evolution |
| Enrichment | Lookup + inject context |
6. Configuring Dead Letter Queues
| Setting | Detail |
|---|---|
| Max delivery attempts | 5 |
| DLQ topic | Same name + .dlq |
| Retention | 14 days minimum |
| Alerting | Page on DLQ depth > 0 |
| Replay tool | Move from DLQ to main |
7. Implementing Event Replay
| Method | Detail |
|---|---|
| Offset reset | Consumer rewinds to N |
| Time-based | Replay from timestamp |
| Selective | Filter by event_id |
| New consumer group | Fresh read, no interference |
8. Using Event Filtering
Example: CloudEvents filter
filters:
- prefix:
type: com.example.order.
- suffix:
subject: .premium
- exact:
source: /us-east/payments
- all:
- type: order.created
- subject.startsWith: enterprise9. Setting Up Event Acknowledgment
| Mode | Guarantee |
|---|---|
| At-most-once | Auto-ack on receive (may lose) |
| At-least-once | Ack after process (may dup) |
| Exactly-once | Idempotent + txns (complex) |
| Manual ack | App controls offset commit |
10. Configuring Event Batching
| Setting | Trade-off |
|---|---|
| Batch size | Larger = throughput, latency |
| Linger time | Wait to fill batch (5-100ms) |
| Compression | Zstd/snappy in batch |
| Max in-flight | Pipelining cap |