Designing Load Balancer Architecture
1. Understanding Load Balancing Algorithms
| Algorithm | Behavior | Best For |
|---|
| Round Robin | Cycle through backends | Equal-capacity nodes |
| Least Connections | Send to backend with fewest active conns | Long-lived requests |
| Least Response Time | Lowest p50 latency | Heterogeneous backends |
| IP Hash | Hash client IP → backend | Sticky sessions w/o cookies |
| Random (P2C) | Pick 2 random, choose less loaded | Scales well, low coordination |
| Consistent Hash | Stable mapping by key | Caches, sharded backends |
2. Understanding Weighted Load Balancing
| Variant | Use Case |
|---|
| Weighted Round Robin | Backends with different CPU/mem |
| Weighted Least Conn | Mixed instance sizes |
| Weighted Random | Canary: 95% old, 5% new |
| Dynamic weights | EWMA of latency adjusts weight |
3. Designing Layer 4 Load Balancing
| Property | L4 |
|---|
| Layer | TCP/UDP |
| Inspection | IP + port only |
| Latency | Sub-ms |
| Throughput | Millions of pps |
| Examples | AWS NLB, IPVS, HAProxy TCP, MetalLB |
| Best For | High-throughput TCP, gRPC, databases |
4. Designing Layer 7 Load Balancing
| Property | L7 |
|---|
| Layer | HTTP/HTTPS, gRPC |
| Inspection | Headers, paths, cookies |
| Features | Routing, rewrites, auth, WAF |
| Latency overhead | 1–5ms |
| Examples | AWS ALB, Envoy, NGINX, Traefik, HAProxy HTTP |
5. Configuring Health Checks and Failover
| Type | Description | Trade-off |
|---|
| TCP | Open socket on port | Doesn't detect app health |
| HTTP | GET /health expecting 200 | Most common |
| Active | LB initiates check periodically | Standard |
| Passive | Detect via real traffic errors | Faster but noisier |
| Deep | /health checks DB, cache, deps | Risk of cascading marks |
Note: Use shallow checks for liveness, deep checks separately. Consecutive failures (e.g., 3) before marking unhealthy avoids flapping.
6. Designing Session Persistence
| Method | Mechanism | Trade-off |
|---|
| Cookie-based | LB sets sticky cookie | Works behind NAT |
| IP affinity | Hash client IP | Breaks behind shared NAT |
| Application cookie | App emits session cookie | Tied to app version |
| External session | Redis/DB-backed sessions | Stateless app, slight latency |
7. Using DNS-Based Load Balancing
| Policy | Behavior |
|---|
| Round-robin DNS | Multiple A records |
| GeoDNS | Route by client geography |
| Latency-based | Route to lowest-RTT region |
| Weighted | Traffic split for canaries |
| Failover | Health-check + secondary record |
Warning: DNS TTLs cached by clients/resolvers. Failover via DNS can take minutes; pair with VIP/anycast for fast failover.
8. Implementing Global Server Load Balancing (GSLB)
| Feature | Description |
|---|
| Anycast | Same IP advertised from many sites; BGP routes to nearest |
| GeoDNS | DNS responses vary by client region |
| Health-aware | Drains unhealthy regions |
| Examples | AWS Global Accelerator, Cloudflare, Akamai GTM |
9. Designing SSL/TLS Termination
| Mode | Where TLS Ends | Notes |
|---|
| Termination at LB | LB decrypts; backend HTTP | Centralized certs, easier WAF |
| Passthrough | End-to-end TLS | Backend owns certs; no L7 features |
| Re-encryption | Decrypt + re-encrypt | L7 features + in-transit encryption |
Note: Use TLS 1.3, OCSP stapling, automate cert rotation (ACM, cert-manager).
10. Implementing Rate Limiting at Load Balancer
| Strategy | Description |
|---|
| Per-IP | X req/sec per source IP |
| Per-API key | Tiered quotas per consumer |
| Global | Cap total RPS to backend |
| Burst + sustained | Token bucket: burst capacity + refill |
| Response | HTTP 429 with Retry-After header |
11. Designing Reverse Proxy Architecture
| Capability | Examples |
|---|
| Routing | Path/host/header rules |
| Caching | Static assets, idempotent GETs |
| Compression | gzip, brotli |
| Auth offload | OIDC, JWT validation |
| Software | NGINX, Envoy, HAProxy, Traefik, Caddy |
12. Designing Load Balancer High Availability
Clients ──▶ VIP (anycast/floating) ──▶ Active LB ──▶ Backends
│ │
└───────keepalived──────┘
VRRP heartbeat
│
Standby LB (takes over)
| Pattern | Mechanism |
|---|
| Active-passive | VRRP / keepalived floating VIP |
| Active-active | ECMP + anycast across LB nodes |
| Cloud-managed | AWS ALB/NLB, GCP LB (multi-AZ built-in) |
| DSR (direct server return) | Egress bypasses LB; lower load |