Working with Distributed Debugging
1. Implementing Distributed Breakpoints
| Approach | Detail |
|---|
| Coordinated breakpoints | Pause across services on matching trace id |
| Dynamic instrumentation | Lightrun, Rookout — non-stop snapshot at line |
| Conditional | Trigger when trace.id == X && user == Y |
| Risk | Production halts cause cascading timeouts |
2. Understanding Remote Debugging
| Tool | Detail |
|---|
| JDWP (Java) | -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 |
| delve (Go) | dlv attach <pid> --headless --listen=:2345 |
| debugpy (Python) | python -m debugpy --listen 0.0.0.0:5678 --wait-for-client |
| node --inspect | Chrome DevTools Protocol |
Warning: Never expose remote debug ports to untrusted networks — full code execution.
3. Implementing Request Replaying
| Tool | Detail |
|---|
| GoReplay | Capture and replay HTTP at L7 |
| tcpcopy / tcpdump | Capture traffic |
| Replay buffer | Kafka of all requests |
| Use | Reproduce prod bugs in staging |
4. Implementing Traffic Mirroring
Example: Istio mirroring
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: orders }
spec:
hosts: [orders]
http:
- route:
- destination: { host: orders, subset: v1 }
weight: 100
mirror: { host: orders, subset: v2 }
mirrorPercentage: { value: 10.0 }
| Property | Detail |
|---|
| Fire-and-forget | Mirror response discarded |
| No user impact | Test new version on real traffic |
| Watch out | Side effects (writes) duplicated |
5. Using Distributed Profilers
| Tool | Detail |
|---|
| Pyroscope / Grafana Phlare | Continuous fleet-wide profiling |
| Parca | eBPF-based, language-agnostic |
| Datadog Continuous Profiler | Production-safe, low overhead |
| Output | Aggregated flame graphs across pods |
6. Implementing Memory Dump Analysis
| Step | Tool |
|---|
| Capture (Java) | jcmd <pid> GC.heap_dump /tmp/heap.hprof |
| Analyze | Eclipse MAT, JXRay, VisualVM |
| Look for | Dominator tree, leak suspects, retained sizes |
| Native (Go) | go tool pprof http://svc/debug/pprof/heap |
7. Understanding Heisenbug Detection
| Property | Detail |
|---|
| Definition | Bug disappears when observed (logging, debugger) |
| Causes | Race conditions, timing, optimizer-elided code |
| Detection | ThreadSanitizer, Java Flight Recorder, deterministic replay |
| Prevention | Eliminate shared mutable state, use immutability |
8. Implementing Distributed Assertions
| Approach | Detail |
|---|
| Invariant checks | Periodic background validation (totals, counts) |
| Cross-service | Reconciliation jobs, auditors |
| Soft fail | Log + metric, don't crash on prod |
9. Implementing Testing in Production
| Technique | Detail |
|---|
| Feature flags | Roll out to internal users first |
| Shadow / mirror | Run new code on copy of traffic |
| Canary | Small % real users |
| Synthetic tx | Marked test orders, filtered at boundary |
| Chaos | Injected failures (Gremlin, Litmus) |
10. Using Observability for Debugging
| Workflow | Detail |
|---|
| 1. Detect | Alert from SLO burn |
| 2. Triage | Service map → which dep degraded? |
| 3. Drill | Trace exemplar → slow span → log line |
| 4. Confirm | Profile flame graph, heap, queries |