Implementing Cluster Management
1. Understanding Cluster Architecture
| Topology | Detail |
|---|---|
| Master-worker (K8s) | Control plane + data plane |
| Peer-to-peer (Cassandra) | All nodes equal; gossip |
| Leader-follower | One leader per partition (Kafka, etcd) |
| Sharded (Vitess) | Multiple independent shards |
2. Implementing Node Health Monitoring
| Mechanism | Detail |
|---|---|
| Heartbeat | Periodic ping; miss N → suspect |
| Phi accrual | Adaptive (Cassandra, Akka) |
| Gossip SWIM | Indirect probe via peers |
| K8s NodeStatus | kubelet → API server every 10s |
3. Implementing Cluster Membership Changes
| Operation | Detail |
|---|---|
| Join | Bootstrap from seed; receive state |
| Leave (graceful) | Drain, hand off shards, depart |
| Fail (ungraceful) | Detected → re-replicate |
| Joint consensus (Raft) | Safe membership change without split-brain |
4. Implementing Rolling Cluster Upgrades
| Step | Detail |
|---|---|
| Drain node | Move workloads off |
| Upgrade | One node at a time |
| Validate | Health, version checks |
| Continue | Respect quorum (N/2+1 always up) |
| K8s | kubectl drain + node pool blue/green |
5. Handling Cluster Split and Merge
| Scenario | Mitigation |
|---|---|
| Split-brain | Quorum requirement; STONITH |
| Network partition | Minority side becomes read-only or rejects writes |
| Heal / merge | Reconcile divergent state (CRDT, last-writer-wins, manual) |
| Witness / arbiter | Tie-breaker node in 3rd zone |
6. Implementing Resource Allocation
| Mechanism | Detail |
|---|---|
| Requests / Limits (K8s) | Scheduler bin-packing + cgroup enforce |
| QoS classes | Guaranteed / Burstable / BestEffort |
| Node selectors / affinity | Place on right hardware |
| Taints / tolerations | Reserve nodes for workloads |
| Pod priority | Preempt lower-priority on pressure |
7. Implementing Cluster Autoscaling
| Layer | Tool |
|---|---|
| Pod (HPA) | Scale replicas by CPU/mem/custom |
| Pod (VPA) | Resize requests/limits |
| Node (Cluster Autoscaler) | Add/remove nodes |
| Karpenter | Just-in-time provisioning, picks instance type |
| KEDA | Event-driven scale (queue depth, Kafka lag) |
8. Implementing Cluster State Management
| Store | Detail |
|---|---|
| etcd (K8s) | Raft, strongly consistent |
| Consul | KV + service catalog |
| ZooKeeper | Legacy (Kafka pre-KRaft, Hadoop) |
| GitOps (Argo CD, Flux) | Git as source of truth; reconcile loop |
9. Understanding Cluster Coordination
| Primitive | Use |
|---|---|
| Leader election | etcd / ZK lease |
| Distributed lock | Lease + fencing token |
| Barrier | Wait for N members |
| Service discovery | Consul, etcd, K8s Endpoints |
| Config + secrets | Centralized KV |
10. Implementing Cluster Monitoring and Alerting
| Signal | Tool |
|---|---|
| Node metrics | node_exporter |
| K8s objects | kube-state-metrics |
| Control plane | etcd, apiserver SLO dashboards |
| Alerts | NodeNotReady, EtcdHighLatency, APIServerErrors, PodCrashLooping |
| Dashboards | Grafana K8s mixin, Lens, k9s |