Managing Service Coordination
1. Leader Election Pattern
| Aspect | Detail |
|---|---|
| Purpose | One instance performs role (singleton task) across cluster |
| Mechanism | Distributed lock with TTL or consensus protocol |
| Tools | ZooKeeper, etcd lease, Consul session, K8s lease |
| Use Cases | Cron job, partition owner, primary writer |
Example: K8s Lease Election
lock := &resourcelock.LeaseLock{
LeaseMeta: metav1.ObjectMeta{Name: "scheduler", Namespace: "default"},
Client: client.CoordinationV1(),
LockConfig: resourcelock.ResourceLockConfig{Identity: hostname},
}
leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
Lock: lock, LeaseDuration: 15*time.Second,
RenewDeadline: 10*time.Second, RetryPeriod: 2*time.Second,
Callbacks: leaderelection.LeaderCallbacks{
OnStartedLeading: func(ctx context.Context) { runScheduler(ctx) },
OnStoppedLeading: func() { stopScheduler() },
},
})2. Distributed Lock Pattern
| Property | Required |
|---|---|
| Mutual Exclusion | Only one holder |
| Deadlock-Free | TTL ensures release on holder death |
| Fault-Tolerant | Lock service must be HA |
| Fencing Token | Resists stale-holder writes |
3. Distributed Consensus Pattern
| Algorithm | Description |
|---|---|
| Paxos | Classic; complex; hard to implement correctly |
| Raft | Equivalent to Paxos; designed for understandability |
| Zab | ZooKeeper Atomic Broadcast |
| PBFT | Byzantine fault tolerant |
| Quorum | Majority must agree (e.g., 3 of 5) |
4. Coordination Service Pattern
| Capability | Provided By |
|---|---|
| Distributed Locks | etcd, ZooKeeper, Consul |
| Leader Election | etcd lease, K8s lease |
| Service Discovery | Consul, etcd, ZooKeeper |
| Config Distribution | etcd watch, Consul KV |
| Membership | Hashicorp Serf, ZooKeeper ephemeral nodes |
5. Work Distribution Pattern
| Strategy | Detail |
|---|---|
| Queue-Based | Workers compete on shared queue |
| Hash-Based Sharding | Hash(key) → owner instance |
| Consistent Hashing | Minimize re-sharding on instance change |
| Coordinator-Assigned | Leader assigns work to followers |
6. Task Queue Pattern
| Element | Detail |
|---|---|
| Producer | Enqueues task with payload + priority |
| Worker | Polls queue, executes, acks |
| Visibility Timeout | Task hidden during processing; reappears if not acked |
| DLQ | Failed-too-many-times tasks parked |
| Tools | SQS, RabbitMQ, Celery, Sidekiq, Temporal Activities |
7. Claim Check Pattern
| Aspect | Detail |
|---|---|
| Problem | Large payloads exceed broker message size limits |
| Solution | Store payload in blob store; pass reference (URL/key) in message |
| Consumer | Fetches payload using claim check ID |
| Cleanup | TTL on blob; or consumer deletes after process |
Example: Claim Check with S3
String key = "uploads/" + UUID.randomUUID();
s3.putObject(bucket, key, largePayload);
kafka.send("uploads", new ClaimCheck(key, bucket, payload.length()));
// Consumer:
ClaimCheck cc = record.value();
byte[] data = s3.getObject(cc.bucket(), cc.key());8. Distributed Barrier Pattern
| Aspect | Detail |
|---|---|
| Purpose | N participants wait until all reach barrier, then proceed |
| Implementation | ZooKeeper ephemeral nodes; etcd counter; Redis |
| Use Case | Coordinated rollout, batch phase boundary |
9. Distributed Latch Pattern
| Aspect | Detail |
|---|---|
| Purpose | Wait until counter reaches 0 (one-shot) |
| Use Case | Wait for N events before proceeding |
| Implementation | Atomic counter in Redis/etcd; watcher on 0 |
10. Scheduling Coordinator Pattern
| Aspect | Detail |
|---|---|
| Role | Decides where/when work runs across cluster |
| Considers | Resource availability, locality, priority, fairness |
| Examples | K8s scheduler, YARN, Mesos, Nomad |
| HA | Leader-elected; followers ready to take over |