Implementing Observability Practices
1. Understanding Observability Pillars
| Pillar | Question Answered |
|---|---|
| Metrics | What is broken? (aggregate health) |
| Logs | What happened? (events) |
| Traces | Where in the request flow? (causality) |
| Profiles (4th) | Why is it slow at code level? |
2. Implementing Distributed Tracing
| Step | Detail |
|---|---|
| SDK | OpenTelemetry auto + manual instrumentation |
| Propagation | W3C traceparent everywhere |
| Backend | Tempo / Jaeger / Honeycomb |
3. Collecting Metrics
| Step | Detail |
|---|---|
| Instrument | RED + USE + business |
| Export | OTLP or /metrics |
| Cardinality budget | Define per service |
4. Centralizing Logs
| Step | Detail |
|---|---|
| Stdout JSON | Container-friendly |
| Agent | Fluent Bit / Vector / OTel collector |
| Backend | Loki / OpenSearch / Elastic |
5. Creating Dashboards
| Pattern | Detail |
|---|---|
| Single pane | Metrics + logs + traces (Grafana) |
| Drill-down | Click span → related logs |
| Versioned | Dashboards-as-code (Jsonnet/Terraform) |
6. Setting Up Alerts
| Source | Detail |
|---|---|
| Metrics | SLO burn rate |
| Logs | Error pattern explosion |
| Traces | Anomalous latency |
| Routing | Alertmanager → PagerDuty / Opsgenie |
7. Implementing SLO Monitoring
| Element | Detail |
|---|---|
| Multi-window burn rate | 5m+1h, 1h+6h pairs |
| Tools | Sloth, Pyrra, Nobl9 |
8. Using Correlation IDs
| Field | Detail |
|---|---|
| traceId | Across services |
| requestId | Per HTTP request |
| In all logs | Auto via MDC/context |
9. Implementing Error Tracking
| Tool | Capability |
|---|---|
| Sentry / Rollbar / Bugsnag | Group, dedupe, alert |
| Release tagging | Regression detection |
| Source maps / symbols | Readable stacks |
10. Creating Service Dependency Maps
| Source | Detail |
|---|---|
| Traces | Auto service graph |
| Service mesh | Kiali (Istio) |
| CMDB / Backstage | Declared |
11. Implementing Audit Trails
| Element | Detail |
|---|---|
| What | actor, action, resource, before/after, ts |
| Storage | Append-only / WORM |
| Retention | Per regulation |
12. Using OpenTelemetry for Standardization
| Component | Detail |
|---|---|
| SDKs | Vendor-neutral instrumentation |
| OTLP | Wire protocol (gRPC/HTTP) |
| Collector | Receive → process → export to any backend |
| Semantic conventions | Shared attribute names |