Implementing Distributed Tracing
1. Understanding Distributed Tracing Concepts
| Concept | Detail |
|---|---|
| Trace | End-to-end request path across services |
| Span | Single unit of work with start/end |
| Trace ID | Unique per request (16 bytes / 32 hex) |
| Span ID | Unique per span (8 bytes / 16 hex) |
| Parent Span ID | Builds tree |
| Context propagation | W3C traceparent header |
2. Generating Trace IDs
| Format | Detail |
|---|---|
| W3C | traceparent: 00-{trace-id}-{span-id}-{flags} |
| Generation | SDK assigns at edge if missing |
| Encoding | Lowercase hex |
| Length | 32 hex (trace), 16 hex (span) |
3. Propagating Trace Context
| Header | Use |
|---|---|
| traceparent | W3C ID + parent |
| tracestate | Vendor extensions |
| baggage | Cross-cutting key/values |
| B3 (Zipkin) | X-B3-TraceId, X-B3-SpanId |
4. Creating Spans
Example: OpenTelemetry (Java)
Tracer tracer = GlobalOpenTelemetry.getTracer("orders");
Span span = tracer.spanBuilder("processOrder")
.setAttribute("order.id", id)
.startSpan();
try (var scope = span.makeCurrent()) {
chargePayment(id);
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw e;
} finally {
span.end();
}| Element | Detail |
|---|---|
| Name | Operation: HTTP GET /orders/{id} |
| Kind | SERVER, CLIENT, PRODUCER, CONSUMER, INTERNAL |
| Status | OK, ERROR, UNSET |
5. Using Parent-Child Relationships
| Relation | Detail |
|---|---|
| Parent → Child | Synchronous nested call |
| Follows-from | Async causation (queues) |
| Span links | Connect spans across traces (batches) |
6. Adding Span Tags
| Convention | Examples |
|---|---|
| HTTP | http.method, http.status_code, url.full |
| DB | db.system, db.statement |
| Messaging | messaging.system, messaging.destination |
| Custom | order.id, tenant.id |
| Spec | OpenTelemetry Semantic Conventions |
7. Adding Span Logs
| Element | Detail |
|---|---|
| Span event | Time-stamped record on a span |
| recordException | Captures stack as event |
| Use | Mark stages within a span without new span |
8. Implementing Sampling Strategies
| Strategy | Detail |
|---|---|
| Head-based (probabilistic) | Decide at root; e.g. 1% |
| Rate-limiting | N traces/sec per service |
| Tail-based | Decide after seeing whole trace (errors, slow) |
| Adaptive | Adjust based on volume |
| Tip | Always sample errors + slow traces |
9. Correlating Traces Across Services
| Mechanism | Detail |
|---|---|
| Auto-instrumentation | SDK injects/extracts headers |
| Async | Inject context into message headers |
| Cron / batch | New trace per run; link to parent |
10. Analyzing Trace Data
| Insight | How |
|---|---|
| Latency hotspots | Critical path analysis |
| Error origin | Failing span at deepest level |
| Service map | Auto-derived from traces |
| High-cardinality query | Filter by tag (user, tenant) |
11. Implementing Error Tracing
| Practice | Detail |
|---|---|
| Status ERROR | Mark on exception |
| recordException | Stack trace as event |
| Force-sample | Always keep error traces |
| Link to logs | traceId in log lines |
12. Using Tracing Tools
| Tool | Notes |
|---|---|
| OpenTelemetry | Vendor-neutral SDK + collector (standard) |
| Jaeger | OSS backend |
| Tempo | Grafana, object-storage backed |
| Zipkin | OSS, B3 origin |
| Datadog / Honeycomb / New Relic / Lightstep | Managed APM |