Implementing Distributed Coordination

1. Understanding Coordination Services

ServiceBackendAPI Style
ZooKeeperZabHierarchical znodes
etcdRaftKey-value + watch
ConsulRaftKV + service catalog
ChubbyPaxosInternal Google lock service

2. Implementing Barrier Synchronization

StepAction
1Each participant creates ephemeral child under /barrier
2When child count == N, all may proceed
3Use watches to be notified of count change

3. Implementing Distributed Locks

Example: Redis SET NX lock with token

String token = UUID.randomUUID().toString();
String resp = jedis.set("lock:order:42", token,
        SetParams.setParams().nx().px(10_000));
if ("OK".equals(resp)) {
    try { criticalSection(); }
    finally {
        // Release only if we still own it
        String lua = "if redis.call('get', KEYS[1]) == ARGV[1] then "
                   + "return redis.call('del', KEYS[1]) else return 0 end";
        jedis.eval(lua, List.of("lock:order:42"), List.of(token));
    }
}
ImplementationNotes
Redis SET NX PXSingle-instance; needs Redlock or fencing for safety
RedlockMultiple Redis instances; debated correctness
ZooKeeper recipeEphemeral sequential znode
etcd lease + lockStrong, with revision as fencing token
Warning: Distributed locks are not safe without fencing tokens — GC pauses or network delays can cause two clients to think they hold the lock simultaneously.

4. Implementing Distributed Semaphores

ApproachDetail
N child znodesAcquire = create ephemeral; release = delete
etcd transactionsCAS on counter with bound
RedisLua script with INCR + CAS

5. Implementing Distributed Counters

StrategyTrade-off
Single-key INCRStrong; bottleneck at key
Sharded countersSum N keys on read; scales writes
G-Counter (CRDT)Mergeable; eventually consistent
HyperLogLogApproximate distinct count

6. Implementing Group Membership Management

MechanismDetail
Ephemeral nodesZooKeeper auto-removes on disconnect
Lease + heartbeatetcd / Consul
Gossip membershipSWIM (Serf, Memberlist)

7. Implementing Distributed Queues

BackendPattern
ZooKeeperSequential znodes; consumers watch lowest
Redis StreamsConsumer groups, XREADGROUP, ACK
KafkaPartitioned log + consumer groups
SQS / Pub/SubManaged at-least-once

8. Implementing Configuration Management

PatternDetail
Watch-basedClients subscribe to key changes (etcd watch)
Versioned KVAtomic updates, rollback by revision
Push vs pullPush reduces latency; pull simpler
ToolsConsul KV, Spring Cloud Config, etcd

9. Implementing Distributed Barriers

VariantUse
Entry barrierWait for N participants to arrive
Exit barrierWait for all to finish phase before next
Double barrierBoth entry and exit
Use caseMapReduce phases, batch jobs

10. Understanding Coordination Trade-offs

TradeDetail
Strong vs eventualCoordination = serialization point
Centralized serviceSimplifies semantics; SPOF if not HA
Coordination costRound-trip per op; minimize via batching
Avoid when possibleIdempotency & CRDTs reduce need