Implementing Multi-Datacenter Replication

1. Understanding Multi-DC Patterns

Pattern Description Detail
Active-Passive DR standby DC Failover only
Active-Active Both serve traffic Bidirectional
Stretch Single cluster across DCs Low-latency links

Example: Stretch cluster racks

broker.rack=dc1
min.insync.replicas=2   # replicas across DCs

2. Using Cluster Linking

Aspect Description Detail
Broker-Native No Connect needed Confluent
Offset Preserving Same offsets Seamless failover
Mirror Topics Read-only replica Promote on DR 
kafka-cluster-links --create --link dr-link \
  --config-file link.properties --bootstrap-server dst:9092

3. Configuring Geo-Replication

Aspect Description Detail
Async Cross-region copy Eventual
Compression Reduce WAN bytes zstd
Topic Prefix Source-named Loop safe

Example: WAN compression

src->dst.producer.compression.type=zstd

4. Setting Up Disaster Recovery

Aspect Description Detail
RPO Max data loss window Replication lag
RTO Recovery time Failover speed
Offset Sync Resume consumers Checkpoints

Example: Translate offsets on DR

kafka-mirror-checkpoint --bootstrap-server dst:9092 \
  --group orders --source src --target dst

5. Implementing Failover Strategy

Step Description Detail
Detect Primary DC down Health check
Redirect Clients to DR DNS/LB
Seek Translated offsets Resume

Example: Seek on failover

consumer.seek(partition, translatedOffset);
consumer.poll(Duration.ofMillis(100));

6. Configuring Conflict Resolution

Approach Description Detail
Partitioned Keys Region owns key ranges No conflict
LWW Last write wins Timestamp
CRDT Mergeable state App-level

Example: Region-scoped key

String key = region + ":" + entityId;  // avoid cross-DC clash

7. Monitoring Cross-Cluster Lag

Metric Description Detail
replication-latency WAN delay RPO proxy
Heartbeat Lag Flow health Probe topic
Alert Lag > RPO threshold Page

Example: Lag from heartbeats

kafka-console-consumer.sh --bootstrap-server dst:9092 \
  --topic src.heartbeats --property print.timestamp=true

8. Using Replicator

Aspect Description Detail
Confluent Replicator Connect-based tool Enterprise
Config Sync Topic settings Auto
Filtering Transform/select SMT

Example: Replicator connector

{
  "connector.class": "io.confluent.connect.replicator.ReplicatorSourceConnector",
  "topic.whitelist": "orders",
  "src.kafka.bootstrap.servers": "src:9092"
}

9. Configuring Network Latency Handling

Config Description Detail
request.timeout.ms Higher for WAN Tolerance
retry.backoff.ms Spaced retries Avoid storms
Batching Larger batches Amortize RTT

Example: WAN-tuned producer

request.timeout.ms=60000
linger.ms=100
batch.size=131072

10. Testing Failover Procedures

Aspect Description Detail
Game Day Scheduled drill Validate RTO
Verify Offsets No duplicate/skip Correctness
Failback Return to primary Reverse flow

Example: Verify DR consumer position

kafka-consumer-groups.sh --bootstrap-server dst:9092 \
  --describe --group orders

Related Cheat Sheets

Kafka

Managing Cluster Operations

Kafka

Working with Time-Based Processing

Kafka

Using Kafka CLI Tools

Kafka

Testing Kafka Applications