Implementing Chaos Engineering
1. Injecting Latency Delays
Example: Istio fault injection
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: orders-chaos }
spec:
hosts: [orders]
http:
- fault:
delay:
percentage: { value: 10 }
fixedDelay: 5s
route: [{ destination: { host: orders } }]2. Simulating Service Failures
| Failure | Tool |
|---|---|
| Pod kill | chaos-mesh PodChaos, kube-monkey |
| Network partition | NetworkChaos, Toxiproxy |
| Process kill | SIGKILL injection |
| DNS failure | DNSChaos resolve errors |
| Disk pressure | StressChaos fill /tmp |
3. Implementing Random Error Injection
Example: Envoy fault filter
http_filters:
- name: envoy.filters.http.fault
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault
abort:
percentage: { numerator: 1, denominator: HUNDRED }
http_status: 503
delay:
percentage: { numerator: 5, denominator: HUNDRED }
fixed_delay: 2s4. Using Traffic Shadowing
| Aspect | Detail |
|---|---|
| Purpose | Test new version with real traffic |
| Mirror % | Start at 1-5% |
| Response handling | Discarded, never returned to client |
| Side effects | Risk: writes to real DB → use sandbox |
| Comparison | diff old vs new responses |
5. Configuring Fault Injection Rules
| Scope | Detail |
|---|---|
| User filter | Header x-chaos: true |
| Percentage | Start small (1%) |
| Duration | Time-boxed window |
| Blast radius | One region/cell first |
| Kill switch | Disable in < 30s |
6. Setting Up Circuit Breaker Testing
| Test | Expected |
|---|---|
| Trigger 5 consecutive 5xx | CB opens |
| Subsequent requests | Fast-fail without backend hit |
| After cooldown | Half-open, one trial |
| Success in half-open | CB closes |
| Fallback response | Cached or default |
7. Implementing Timeout Simulation
| Layer | Inject |
|---|---|
| Network | tc netem delay 30s |
| Application | Sleep in handler |
| Proxy | fixedDelay > client timeout |
| DB | pg_sleep(N) injected query |
8. Using Resource Exhaustion Tests
| Resource | Stress Tool |
|---|---|
| CPU | stress-ng --cpu N |
| Memory | stress --vm-bytes |
| Disk I/O | fio |
| File descriptors | Open N sockets |
| Connection pool | Exhaust DB pool |
9. Configuring Partial Outage Scenarios
| Scenario | Validate |
|---|---|
| 1 of 3 AZs down | Quorum maintained, traffic shifts |
| Cache down | Graceful origin fallback |
| Auth down | Cached token validation |
| DB read-replica lag | Stale-read tolerance |
| Region isolation | Failover within RTO |