Implementing Load Balancing
1. Understanding Load Balancing Algorithms
| Algorithm | Behavior | Best For |
|---|---|---|
| Round Robin | Sequential rotation | Homogeneous backends |
| Least Connections | Pick lowest active count | Long-lived connections |
| Least Response Time | Pick fastest avg latency | Heterogeneous capacity |
| Weighted RR | Static weights | Mixed capacity |
| IP / Hash | Sticky by source | Cache locality |
| Random + 2-choice | Pick 2 random, choose lighter | Avoids herd; near-optimal |
| EWMA / P2C | Exponentially-weighted moving avg | Latency-aware (Finagle) |
2. Implementing Weighted Load Balancing
| Use Case | Detail |
|---|---|
| Heterogeneous nodes | Larger instance gets higher weight |
| Canary release | 5% to new version, 95% to old |
| Cross-AZ preference | Same-AZ weight higher |
3. Implementing Consistent Hash Load Balancing
| Property | Detail |
|---|---|
| Mechanism | Hash(key) → ring → node |
| Use case | Cache affinity, session affinity |
| Bounded loads | Bound max load via overflow |
| Examples | Envoy ring_hash, Maglev |
4. Implementing Session Affinity (sticky sessions)
| Method | Detail |
|---|---|
| Cookie | LB sets AWSALB / SERVERID |
| Source IP hash | Same client → same backend |
| Token-based | Hash header (e.g., user id) |
| Trade-off | Hot backends; complicates rolling deploys |
5. Understanding Layer 4 vs Layer 7 Load Balancing
6. Implementing Client-Side Load Balancing
| Library | Detail |
|---|---|
| gRPC name resolver | Pluggable; round-robin / pick-first |
| Spring Cloud LoadBalancer | Replaces Ribbon |
| Finagle | P2C, EWMA |
| Service mesh sidecar | Envoy applies LB locally |
7. Implementing Server-Side Load Balancing
| Type | Examples |
|---|---|
| Hardware | F5, Citrix NetScaler |
| Software | NGINX, HAProxy, Envoy, Traefik |
| Cloud-managed | ALB, NLB, GCLB, Azure LB |
| Anycast (DNS/BGP) | Cloudflare, Fastly |
8. Handling Load Balancer Health Checks
| Knob | Tuning |
|---|---|
| Interval | 5-30s |
| Timeout | 2-5s |
| Healthy threshold | 2-3 consecutive successes |
| Unhealthy threshold | 2-3 consecutive failures |
| Path | Lightweight, no deps |
9. Implementing Dynamic Load Balancing
| Signal | Adjustment |
|---|---|
| Backend latency | Reduce weight on slow nodes |
| Active connections | Spread to lighter nodes |
| Auto-scaling events | Update pool, drain old |
| Outlier detection | Eject + retry (Envoy) |
10. Understanding Load Shedding Strategies
| Strategy | Detail |
|---|---|
| Reject new requests | HTTP 503 + Retry-After |
| Priority shedding | Drop low-priority first |
| Adaptive concurrency | Limit based on observed latency (Netflix concurrency-limits) |
| Load-aware routing | Route around saturated nodes |