Integrating with Distributed Tracing

1. Configuring OpenTelemetry Integration

Example: OTel collector config

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch:
  resource:
    attributes:
      - key: service.name
        value: api-gateway
        action: upsert
exporters:
  otlp:
    endpoint: tempo:4317
    tls: { insecure: true }
service:
  pipelines:
    traces: { receivers: [otlp], processors: [batch, resource], exporters: [otlp] }
ComponentPurpose
SDKApp instrumentation
CollectorReceive, process, export
OTLPStandard wire protocol
BackendTempo, Jaeger, X-Ray, Honeycomb

2. Using Jaeger Tracing

AspectDetail
Agent port6831 UDP (compact thrift)
Collector14250 gRPC, 14268 HTTP
Query UI16686
StorageCassandra, Elasticsearch, Badger

3. Implementing Zipkin Integration

SettingValue
Endpoint/api/v2/spans
FormatJSON v2, Protobuf
HeadersB3 (x-b3-traceid)
B3-Singleb3: trace-span-sample

4. Setting Up Trace Context Propagation

FormatHeader
W3C Trace Contexttraceparent, tracestate
B3 (Zipkin)X-B3-TraceId, X-B3-SpanId
AWS X-RayX-Amzn-Trace-Id
Jaegeruber-trace-id
Baggagebaggage: key=val,...

5. Using Trace IDs and Span IDs

Example: traceparent header

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              ver-trace-id (32 hex)            -span-id (16 hex) -flags

# flags: 01 = sampled, 00 = not sampled

6. Configuring Sampling Rates

StrategyDetail
Always on100% (dev/staging)
Probability1-10% prod
Rate limitedN traces/sec/service
AdaptiveAuto-tune to budget
Tail-basedSample errors + slow @ 100%

7. Implementing Custom Spans

Example: Custom span (Java)

Span span = tracer.spanBuilder("authorize")
  .setAttribute("tenant.id", tenantId)
  .setAttribute("user.id", userId)
  .startSpan();
try (Scope s = span.makeCurrent()) {
  return authzClient.check(action, resource);
} catch (Exception e) {
  span.recordException(e);
  span.setStatus(StatusCode.ERROR);
  throw e;
} finally {
  span.end();
}

8. Setting Up Service Maps

SourceTool
Span-derivedTempo metrics-generator
Service meshKiali (Istio)
eBPFPixie, Cilium Hubble
APMDatadog, New Relic, Dynatrace

9. Using Distributed Tracing Headers (W3C)

HeaderFormat
traceparent00-traceid-spanid-flags
tracestateVendor-specific KV (congo=t61rcWkgMzE)
baggageApp context propagation

10. Configuring Trace Exporters

ExporterUse
OTLPStandard, preferred
JaegerLegacy thrift/gRPC
ZipkinHTTP JSON
ConsoleDebug stdout
MultipleFan-out via collector