Implementing Observability Patterns
1. Distributed Tracing Pattern
| Concept | Detail |
|---|---|
| Trace | End-to-end request across services |
| Span | Single unit of work; has trace ID, span ID, parent ID |
| Context Propagation | W3C Trace Context (traceparent header) |
| Standards | OpenTelemetry (OTel) — SDKs in all languages |
| Backends | Jaeger, Tempo, Zipkin, Honeycomb, Datadog |
Example: OTel Java Auto-Instrumentation
java -javaagent:opentelemetry-javaagent.jar \
-Dotel.service.name=order-service \
-Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
-jar app.jar2. Correlation ID Pattern
| Aspect | Detail |
|---|---|
| Definition | Single ID stamped at edge; propagated through all services |
| Header | X-Correlation-ID or traceparent trace ID |
| Usage | Logged on every line; included in error responses |
| Benefit | Tie all logs/traces/events for a single business transaction |
3. Log Aggregation Pattern
| Component | Tools |
|---|---|
| Collection | Fluent Bit, Vector, Filebeat, OTel Collector |
| Transport | Kafka, OTLP, syslog |
| Storage | Loki, Elasticsearch, OpenSearch, ClickHouse |
| Query / UI | Grafana, Kibana, Datadog, Splunk |
| Format | Structured JSON; include trace_id, service, level |
4. Application Metrics Pattern
| Type | Use |
|---|---|
| Counter | Monotonic count (requests, errors) |
| Gauge | Snapshot value (queue depth, memory) |
| Histogram | Distribution (request latency) |
| Summary | Pre-computed quantiles |
| RED | Rate, Errors, Duration — for services |
| USE | Utilization, Saturation, Errors — for resources |
| Tools | Prometheus, OTel, Micrometer, Datadog |
5. Health Check API Pattern
| Endpoint | Purpose |
|---|---|
| /health/live | Process alive (binary) |
| /health/ready | Ready for traffic (deps check) |
| /health/startup | Boot complete |
| /health | Aggregate (for humans) |
Example: Health Response
{
"status": "UP",
"checks": {
"db": { "status": "UP", "latencyMs": 12 },
"kafka": { "status": "UP" },
"redis": { "status": "DEGRADED", "message": "high latency" }
}
}6. Exception Tracking Pattern
| Capability | Detail |
|---|---|
| Auto-Capture | Unhandled exceptions reported with stack + context |
| Grouping | Same fingerprint deduplicated |
| Release Tracking | "Introduced in v2.3.0" |
| Tools | Sentry, Rollbar, Bugsnag, Datadog Error Tracking |
7. Audit Logging Pattern
| Field | Detail |
|---|---|
| who | Authenticated principal |
| what | Action performed |
| when | Timestamp (UTC, ISO 8601) |
| where | Source IP, service |
| target | Resource ID |
| result | SUCCESS / DENIED / FAILED |
| Storage | Append-only, retention per compliance (often years) |
8. Distributed Context Propagation Pattern
| Standard | Detail |
|---|---|
| W3C Trace Context | traceparent, tracestate headers |
| W3C Baggage | Arbitrary key-value pairs (tenant, user, feature flag) |
| B3 Headers | Legacy Zipkin format |
| OTel Context | API for cross-cutting context |
9. Service Dependency Mapping Pattern
| Source | Output |
|---|---|
| Trace Data | Auto-derived service graph |
| Service Mesh | Kiali, Linkerd dashboard |
| Static Analysis | Code scan for HTTP/gRPC clients |
| Use | Impact analysis, blast radius, capacity planning |
10. Span Context Pattern
| Field | Detail |
|---|---|
| trace_id | 128-bit; same across whole trace |
| span_id | 64-bit; unique per span |
| parent_span_id | Reference to caller span |
| trace_flags | Sampled bit, debug bit |
| attributes | Key-value tags (http.status, db.statement) |
| events | Timestamped log lines within span |