Implementing Distributed Tracing

1. Using OpenTelemetry SDK

ComponentPurpose
SDKTracer/Meter providers
ExporterOTLP, Jaeger, Zipkin
Instrumentationotelgrpc for gRPC
PropagatorW3C TraceContext + Baggage

2. Creating Trace Provider

Example: OTLP exporter + provider

exp, _ := otlptracegrpc.New(ctx)
tp := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exp),
    sdktrace.WithResource(resource.NewSchemaless(
        semconv.ServiceName("user-service"),
    )),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.TraceContext{})

3. Instrumenting Client with Interceptor

Example: Client otelgrpc

import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
conn, _ := grpc.NewClient(addr,
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
    grpc.WithTransportCredentials(creds),
)

4. Instrumenting Server with Interceptor

Example: Server otelgrpc

srv := grpc.NewServer(grpc.StatsHandler(otelgrpc.NewServerHandler()))
NoteDetail
Modern APIUse StatsHandler; interceptor APIs DEPRECATED in otelgrpc

5. Creating Spans

Example: Manual span

ctx, span := otel.Tracer("user").Start(ctx, "LoadFromDB")
defer span.End()

6. Adding Span Attributes

Example: Set attrs

span.SetAttributes(
    attribute.String("user.id", id),
    attribute.Int("db.rows", n),
)
ConventionDetail
Semantic conventionsUse semconv constants where possible
CardinalityAvoid unbounded values (e.g. UUIDs as attr keys)

7. Recording Span Events

Example: Event & error

span.AddEvent("cache.miss", trace.WithAttributes(attribute.String("key", k)))
if err != nil {
    span.RecordError(err)
    span.SetStatus(otelcodes.Error, err.Error())
}

8. Propagating Trace Context

HeaderPurpose
traceparentW3C trace context
tracestateVendor-specific state
baggageCross-service key-value

9. Exporting Traces

ExporterUse
OTLP/gRPCStandard — to OpenTelemetry Collector
Jaeger / ZipkinDirect (legacy)
stdoutLocal debugging

10. Correlating Logs and Traces

Example: Add trace id to log

sc := trace.SpanContextFromContext(ctx)
slog.InfoContext(ctx, "processed",
    slog.String("trace_id", sc.TraceID().String()),
    slog.String("span_id", sc.SpanID().String()),
)