Implementing Distributed Logging
1. Understanding Structured Logging
Example: JSON structured log
{
"ts": "2026-05-15T10:23:45.123Z",
"level": "INFO",
"service": "order-svc",
"trace_id": "9f2e1c...",
"span_id": "a8b3...",
"user_id": "u-42",
"event": "order.created",
"order_id": "o-7",
"duration_ms": 47
}| Field | Purpose |
|---|---|
| ts | ISO 8601 UTC timestamp |
| level | DEBUG / INFO / WARN / ERROR |
| service / version | Source identification |
| trace_id / span_id | Correlate with traces |
| event / message | Machine + human readable |
2. Implementing Correlation IDs (trace IDs)
| Mechanism | Detail |
|---|---|
| W3C traceparent | traceparent: 00-<trace-id>-<span-id>-01 |
| B3 (Zipkin) | X-B3-TraceId, X-B3-SpanId |
| Generation | Edge gateway creates, propagates downstream |
| MDC (logback) | Per-thread context binding |
3. Implementing Log Aggregation
| Pipeline | Detail |
|---|---|
| Collect | Fluent Bit, Vector, Filebeat, Promtail |
| Transport | Kafka, Kinesis |
| Index / store | Elasticsearch/OpenSearch, Loki, Splunk |
| Query | Kibana, Grafana, Splunk SPL |
4. Implementing Centralized Logging
| Stack | Components |
|---|---|
| ELK / OpenSearch | Logstash → ES → Kibana |
| Loki + Promtail + Grafana | Lower cost, label-based |
| Splunk | Enterprise, broad indexing |
| Datadog / New Relic | Managed SaaS |
5. Understanding Log Levels
| Level | Use |
|---|---|
| TRACE | Verbose, dev only |
| DEBUG | Diagnostic; off in prod usually |
| INFO | Significant business events |
| WARN | Recoverable issue, attention soon |
| ERROR | Action failed; needs investigation |
| FATAL | Process unable to continue |
6. Implementing Log Sampling
| Strategy | Detail |
|---|---|
| Head sampling | Decide at request entry |
| Tail sampling | Decide after observing whole trace; keep errors |
| Rate-limited | N per second per logger |
| Adaptive | Sample less when traffic high |
7. Implementing Log Retention Policies
| Tier | Retention | Storage |
|---|---|---|
| Hot (search) | 7-30 days | SSD ES/OpenSearch |
| Warm | 30-90 days | HDD nodes |
| Cold / archive | 1-7 years | S3 Glacier, IA |
| Compliance | Per regulation (SOX 7yr, GDPR varies) | — |
8. Handling Log Shipping and Buffering
| Concern | Mitigation |
|---|---|
| Backpressure on app | Async appender + bounded queue |
| Loss on crash | Local file buffer + retry |
| Network partition | Persistent on-disk queue (Vector) |
| Spikes | Kafka buffer, downstream rate limit |
9. Implementing Log Parsing and Indexing
| Approach | Detail |
|---|---|
| Native JSON | No parsing; ES indexes fields |
| Grok patterns | Regex extraction (Logstash) |
| VRL (Vector) | Programmatic transforms |
| Cardinality control | Avoid high-cardinality labels in Loki/Prom |
10. Understanding Logging Best Practices
| Practice | Detail |
|---|---|
| Structured (JSON) | Machine parseable |
| Include trace ids | Cross-reference with traces |
| No PII / secrets | Mask, redact, or reject |
| Use levels correctly | Avoid INFO-spam |
| Async appender | Don't block request thread |
| UTC timestamps | Timezone-free |