Integrating with Distributed Tracing
1. Configuring OpenTelemetry Integration
Example: OTel collector config
receivers:
otlp:
protocols:
grpc: { endpoint: 0.0.0.0:4317 }
http: { endpoint: 0.0.0.0:4318 }
processors:
batch:
resource:
attributes:
- key: service.name
value: api-gateway
action: upsert
exporters:
otlp:
endpoint: tempo:4317
tls: { insecure: true }
service:
pipelines:
traces: { receivers: [otlp], processors: [batch, resource], exporters: [otlp] }
| Component | Purpose |
|---|---|
| SDK | App instrumentation |
| Collector | Receive, process, export |
| OTLP | Standard wire protocol |
| Backend | Tempo, Jaeger, X-Ray, Honeycomb |
2. Using Jaeger Tracing
| Aspect | Detail |
|---|---|
| Agent port | 6831 UDP (compact thrift) |
| Collector | 14250 gRPC, 14268 HTTP |
| Query UI | 16686 |
| Storage | Cassandra, Elasticsearch, Badger |
3. Implementing Zipkin Integration
| Setting | Value |
|---|---|
| Endpoint | /api/v2/spans |
| Format | JSON v2, Protobuf |
| Headers | B3 (x-b3-traceid) |
| B3-Single | b3: trace-span-sample |
4. Setting Up Trace Context Propagation
| Format | Header |
|---|---|
| W3C Trace Context | traceparent, tracestate |
| B3 (Zipkin) | X-B3-TraceId, X-B3-SpanId |
| AWS X-Ray | X-Amzn-Trace-Id |
| Jaeger | uber-trace-id |
| Baggage | baggage: key=val,... |
5. Using Trace IDs and Span IDs
Example: traceparent header
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
ver-trace-id (32 hex) -span-id (16 hex) -flags
# flags: 01 = sampled, 00 = not sampled
6. Configuring Sampling Rates
| Strategy | Detail |
|---|---|
| Always on | 100% (dev/staging) |
| Probability | 1-10% prod |
| Rate limited | N traces/sec/service |
| Adaptive | Auto-tune to budget |
| Tail-based | Sample errors + slow @ 100% |
7. Implementing Custom Spans
Example: Custom span (Java)
Span span = tracer.spanBuilder("authorize")
.setAttribute("tenant.id", tenantId)
.setAttribute("user.id", userId)
.startSpan();
try (Scope s = span.makeCurrent()) {
return authzClient.check(action, resource);
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw e;
} finally {
span.end();
}
8. Setting Up Service Maps
| Source | Tool |
|---|---|
| Span-derived | Tempo metrics-generator |
| Service mesh | Kiali (Istio) |
| eBPF | Pixie, Cilium Hubble |
| APM | Datadog, New Relic, Dynatrace |
9. Using Distributed Tracing Headers (W3C)
| Header | Format |
|---|---|
traceparent | 00-traceid-spanid-flags |
tracestate | Vendor-specific KV (congo=t61rcWkgMzE) |
baggage | App context propagation |
10. Configuring Trace Exporters
| Exporter | Use |
|---|---|
| OTLP | Standard, preferred |
| Jaeger | Legacy thrift/gRPC |
| Zipkin | HTTP JSON |
| Console | Debug stdout |
| Multiple | Fan-out via collector |