Implementing Load Balancing
1. Understanding WebSocket Load Balancing Challenges
| Challenge | Detail |
|---|---|
| Long-lived | Conn pinned to backend; round-robin once |
| Upgrade required | LB must support HTTP/1.1 Upgrade |
| Idle timeouts | LB may drop "idle" sockets |
| Uneven load | Old nodes accumulate connections |
2. Using Sticky Sessions
| Use Case | Detail |
|---|---|
| Local in-memory state | Required |
| Redis-backed state | Not required |
| Auth-cookie based | Cookie hash for pinning |
3. Configuring Layer 4 Load Balancers
| LB | Notes |
|---|---|
| AWS NLB | TCP/TLS passthrough; preserves source IP via PROXY proto |
| GCP NLB | Similar to NLB |
HAProxy mode tcp | Fast, simple |
| Envoy TCP proxy | Modern, configurable |
4. Configuring Layer 7 Load Balancers
Example: Nginx upstream
upstream ws_backend {
ip_hash;
server 10.0.0.1:8080;
server 10.0.0.2:8080;
}
server {
listen 443 ssl;
location /ws {
proxy_pass http://ws_backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
}| Directive | Purpose |
|---|---|
proxy_http_version 1.1 | Required for Upgrade |
Upgrade / Connection | Forward upgrade |
proxy_read_timeout | Avoid idle drop |
5. Implementing Client-Side Load Balancing
| Approach | Detail |
|---|---|
| DNS round-robin | Multiple A records |
| Discovery endpoint | HTTP returns WS host list |
| Geo-pick | Closest edge |
| Failover list | Try N hosts on connect failure |
6. Using Connection Draining
| Phase | Detail |
|---|---|
| Drain start | LB stops sending new conns |
| Notify clients | Close 1001 in waves |
| Grace period | e.g. 30s |
| Forced close | Terminate after grace |
7. Handling Load Balancer Health Checks
| Type | Detail |
|---|---|
HTTP /healthz | Liveness + readiness |
| TCP connect | Basic; doesn't test app |
| WS ping endpoint | Full path validation |
8. Implementing Geographic Routing
| Tool | Detail |
|---|---|
| AWS Route 53 latency | DNS-based geo |
| Cloudflare | Anycast + edge WS |
| Fly.io / regions | Closest region routing |
9. Monitoring Connection Distribution
| Metric | Threshold |
|---|---|
| Conns / node | ±20% from mean |
| CPU / node | < 70% sustained |
| FD usage | < 80% ulimit |
| Conn age histogram | Detect old-node accumulation |
10. Testing Failover Scenarios
| Test | Detail |
|---|---|
| Drain one node | Verify graceful reconnect to others |
| Kill node | 1006 + reconnect distribution |
| LB restart | Brief outage tolerance |
| DNS failure | Cache TTL behavior |