Implementing Distributed Tracing
1. Understanding Trace Context Propagation
| Standard | Header |
|---|---|
| W3C Trace Context | traceparent, tracestate |
| B3 (single) | b3: traceId-spanId-sampled-parentId |
| B3 (multi) | X-B3-TraceId, X-B3-SpanId, X-B3-Sampled |
| Jaeger | uber-trace-id |
| Baggage | baggage: key=value,... for cross-cutting metadata |
2. Implementing Span Creation
Example: OpenTelemetry Java
Tracer tracer = GlobalOpenTelemetry.getTracer("order-svc");
Span span = tracer.spanBuilder("createOrder")
.setAttribute("user.id", userId)
.setSpanKind(SpanKind.SERVER)
.startSpan();
try (Scope s = span.makeCurrent()) {
span.addEvent("validation.started");
Order o = createOrder(req);
span.setAttribute("order.id", o.id);
return o;
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR, e.getMessage());
throw e;
} finally {
span.end();
}| Span Field | Detail |
|---|---|
| Name | Operation: HTTP GET /orders/:id |
| Kind | SERVER, CLIENT, PRODUCER, CONSUMER, INTERNAL |
| Attributes | Key-value tags |
| Events | Time-stamped log entries |
| Status | OK / ERROR + message |
3. Implementing Trace Sampling Strategies
| Strategy | Detail |
|---|---|
| Always-on | 100% — expensive |
| Probabilistic head | X% sampled at root |
| Rate-limited | N per second |
| Tail-based | Decide after seeing whole trace; keep errors / slow |
| Adaptive | Per-service rate based on volume |
4. Working with OpenTelemetry
| Component | Purpose |
|---|---|
| SDK | Per-language instrumentation |
| Auto-instrumentation | Java agent, Python opentelemetry-instrument |
| OTLP | Wire protocol (gRPC/HTTP) |
| Collector | Receive, process, export |
| Backends | Jaeger, Tempo, Zipkin, vendor SaaS |
| Signals | Traces, metrics, logs unified |
5. Implementing Baggage for Context
| Property | Detail |
|---|---|
| Definition | Key-value metadata propagated across services |
| Use cases | Tenant id, feature flag, A/B variant |
| Cost | Adds bytes to every request header |
| Caution | Avoid PII; bound size |
6. Understanding Trace Visualization
| View | Insight |
|---|---|
| Waterfall / Gantt | Span timing, parallelism |
| Service map / graph | Topology, error rates |
| Flame graph | Hot path identification |
| Comparison | Diff against baseline |
7. Implementing Service Dependency Mapping
| Source | Detail |
|---|---|
| Trace aggregation | Build graph from spans |
| Service mesh | Kiali (Istio), Linkerd Viz |
| eBPF (Cilium) | Network-level dependency map |
8. Handling Trace Storage and Querying
| Backend | Storage |
|---|---|
| Jaeger | Cassandra, ES, Badger |
| Tempo | S3 / GCS object store |
| Zipkin | ES, MySQL, Cassandra |
| Indexing | By trace id; secondary by service+operation+tags |
9. Implementing Critical Path Analysis
| Step | Action |
|---|---|
| 1 | Find longest path through span tree |
| 2 | Identify dominant span(s) |
| 3 | Optimize or parallelize hottest segment |
10. Understanding Distributed Tracing Trade-offs
| Pro | Con |
|---|---|
| End-to-end latency visibility | Header overhead per request |
| Root cause analysis | Storage cost grows with traffic |
| Service map auto-discovery | Sampling loses outliers (mitigated by tail sampling) |