Implementing Distributed Tracing
1. Using OpenTelemetry SDK
| Component | Purpose |
|---|---|
| SDK | Tracer/Meter providers |
| Exporter | OTLP, Jaeger, Zipkin |
| Instrumentation | otelgrpc for gRPC |
| Propagator | W3C TraceContext + Baggage |
2. Creating Trace Provider
Example: OTLP exporter + provider
exp, _ := otlptracegrpc.New(ctx)
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp),
sdktrace.WithResource(resource.NewSchemaless(
semconv.ServiceName("user-service"),
)),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.TraceContext{})
3. Instrumenting Client with Interceptor
Example: Client otelgrpc
import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
conn, _ := grpc.NewClient(addr,
grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
grpc.WithTransportCredentials(creds),
)
4. Instrumenting Server with Interceptor
| Note | Detail |
|---|---|
| Modern API | Use StatsHandler; interceptor APIs DEPRECATED in otelgrpc |
5. Creating Spans
6. Adding Span Attributes
Example: Set attrs
span.SetAttributes(
attribute.String("user.id", id),
attribute.Int("db.rows", n),
)
| Convention | Detail |
|---|---|
| Semantic conventions | Use semconv constants where possible |
| Cardinality | Avoid unbounded values (e.g. UUIDs as attr keys) |
7. Recording Span Events
Example: Event & error
span.AddEvent("cache.miss", trace.WithAttributes(attribute.String("key", k)))
if err != nil {
span.RecordError(err)
span.SetStatus(otelcodes.Error, err.Error())
}
8. Propagating Trace Context
| Header | Purpose |
|---|---|
traceparent | W3C trace context |
tracestate | Vendor-specific state |
baggage | Cross-service key-value |
9. Exporting Traces
| Exporter | Use |
|---|---|
| OTLP/gRPC | Standard — to OpenTelemetry Collector |
| Jaeger / Zipkin | Direct (legacy) |
| stdout | Local debugging |