Managing Data in Microservices
1. Database per Service Pattern
| Rule | Detail |
|---|
| Ownership | Each service exclusively owns its DB schema |
| Access | Other services use APIs/events; never direct SQL |
| Tech Choice | Each service picks fit-for-purpose DB (polyglot) |
| Cost | No cross-service joins; eventual consistency |
2. Shared Database Anti-Pattern
| Symptom | Consequence |
|---|
| Multiple services write same tables | Schema changes block all teams |
| Hidden coupling via SQL | Refactor breaks other services silently |
| Lock contention | Performance suffers across services |
| Single point of failure | DB outage = all services down |
Warning: A shared database is the #1 way "microservices" devolve into a distributed monolith.
3. Polyglot Persistence Pattern
| Workload | Recommended Store |
|---|
| Transactional CRUD | PostgreSQL, MySQL |
| Document / flexible schema | MongoDB, DynamoDB |
| Cache / session | Redis, Memcached |
| Search | Elasticsearch, OpenSearch |
| Time-series | InfluxDB, TimescaleDB |
| Graph | Neo4j, Neptune |
| Wide-column / scale | Cassandra, ScyllaDB, Bigtable |
| Event log | Kafka, EventStoreDB |
4. CQRS Pattern
| Side | Responsibility |
|---|
| Command Side | Validates + executes state changes; writes to write model |
| Query Side | Optimized read models; denormalized for queries |
| Sync | Events flow from write to read model (eventually consistent) |
| Pros | Independent scaling, optimized schemas, complex query support |
| Cons | Complexity; eventual consistency UX |
5. Materialized View Pattern
| Aspect | Detail |
|---|
| Definition | Pre-computed read-optimized view kept in sync via events |
| Storage | Owned by query service (Redis, Elasticsearch, RDBMS) |
| Sync | Subscribed to source events; updated on each event |
| Rebuild | Replay event stream from beginning |
Example: Customer Order Summary View
@KafkaListener(topics = "orders.placed")
public void onOrderPlaced(OrderPlaced e) {
summaryRepo.upsert(e.customerId(), s -> {
s.orderCount++;
s.lifetimeValue = s.lifetimeValue.add(e.total());
s.lastOrderAt = e.occurredAt();
});
}
6. Data Replication Pattern
| Type | Mechanism |
|---|
| Event-Driven Replica | Subscribe to source events, maintain local copy |
| CDC Replica | Stream DB log changes to consumers |
| Snapshot + Delta | Initial bulk load + ongoing CDC |
| DB-Level Replication | Native primary→replica (read replicas only) |
7. Transaction Log Tailing Pattern
| Aspect | Detail |
|---|
| Source | Database WAL/binlog |
| Tools | Debezium, AWS DMS, Maxwell |
| Output | Kafka topic of row-level change events |
| Use Case | Outbox-less event publishing, replication |
8. Change Data Capture Pattern
| Aspect | Detail |
|---|
| Definition | Capture row-level INSERT/UPDATE/DELETE as events |
| Methods | Log-based (preferred), trigger-based, query-based |
| Use Cases | Sync to data warehouse, search index, cache, downstream services |
| Caveat | Couples consumers to internal schema; prefer Outbox for domain events |
9. Data Mesh Pattern
| Principle | Detail |
|---|
| Domain-Owned Data | Domain teams own their analytical data products |
| Data as Product | Discoverable, addressable, trustworthy, self-describing |
| Self-Serve Platform | Central platform team provides infra primitives |
| Federated Governance | Org-wide standards; domain autonomy on impl |
10. Data Lake Pattern
| Aspect | Detail |
|---|
| Storage | Raw data in object store (S3, GCS, ADLS) |
| Format | Parquet, ORC, Iceberg, Delta Lake, Hudi |
| Query | Athena, BigQuery, Spark, Trino, Presto |
| Pattern | Services emit events → land in lake → analytics, ML |