Monitoring and Logging
1. Logging Request and Response Details
| Field | Always Log |
|---|---|
| Timestamp (ISO 8601) | Yes |
| Method, path, query (sanitized) | Yes |
| Status code | Yes |
| Duration (ms) | Yes |
| userId / clientId | Yes |
| traceId | Yes |
| Request body | Sample / redact PII |
| Response body | Errors only by default |
2. Using Correlation IDs
Example: Correlation ID Propagation
// Inbound: read or generate
String traceId = req.getHeader("X-Request-ID");
if (traceId == null) traceId = UUID.randomUUID().toString();
MDC.put("traceId", traceId);
// Outbound: forward to downstream services
restTemplate.getForEntity(url, ..., headers -> {
headers.set("X-Request-ID", MDC.get("traceId"));
});
// Response: echo back
res.setHeader("X-Request-ID", traceId);3. Monitoring Response Times
| Metric | Tool |
|---|---|
| p50, p95, p99 latency | Prometheus histogram |
| Per-endpoint breakdown | Tag by route |
| Slowest queries | APM trace sampling |
| SLO tracking | Error budget burn |
4. Monitoring Error Rates
| Bucket | Track Separately |
|---|---|
| 4xx (client errors) | Indicates API misuse / bugs |
| 5xx (server errors) | Page on threshold |
| By endpoint | Identify hotspots |
| By dependency | Upstream failure detection |
5. Implementing Health Check Endpoints
| Endpoint | Purpose |
|---|---|
GET /health/live | Process is running (200 always when up) |
GET /health/ready | Ready for traffic (DB connected, caches warm) |
GET /health/startup | Initial bootstrap done (Kubernetes startupProbe) |
Example: Readiness Response
{
"status": "UP",
"components": {
"db": {"status": "UP", "latencyMs": 4},
"redis": {"status": "UP", "latencyMs": 1},
"kafka": {"status": "DEGRADED", "message": "1 of 3 brokers down"}
}
}6. Monitoring API Usage Patterns
| Metric | Insight |
|---|---|
| Requests per endpoint | Most-used features |
| Per-client breakdown | Top consumers |
| Geographic distribution | CDN / region planning |
| Time-of-day patterns | Capacity planning |
7. Using Application Performance Monitoring
| Tool | Strength |
|---|---|
| Datadog | Unified metrics + traces + logs |
| New Relic | Strong APM, AI insights |
| Honeycomb | High-cardinality observability |
| OpenTelemetry | Open standard, vendor-neutral |
| Grafana + Prometheus + Loki + Tempo | Open-source stack |
8. Implementing Structured Logging
Example: JSON Log Entry
{
"timestamp": "2026-05-15T10:30:00.123Z",
"level": "INFO",
"logger": "com.example.UserController",
"message": "User created",
"traceId": "abc-123",
"userId": "42",
"method": "POST",
"path": "/users",
"status": 201,
"durationMs": 87
}9. Setting Up Alerts
| Condition | Severity |
|---|---|
| 5xx rate > 1% for 5 min | Page |
| p99 latency > SLO | Warn |
| Error budget > 50% burn (7d) | Page |
| Auth failure spike | Security alert |
| Disk > 85% | Warn |
10. Logging Security Events
| Event | Required Fields |
|---|---|
| Login success/failure | userId, IP, userAgent, timestamp |
| Permission denied | userId, resource, action |
| Token issued / revoked | userId, tokenId, source |
| Admin action | actorId, target, change diff |
| Storage | Tamper-evident, retained per compliance |
11. Implementing Log Retention Policies
| Log Type | Retention |
|---|---|
| Application (DEBUG/INFO) | 7-30 days |
| Access logs | 30-90 days |
| Security/audit | 1-7 years (compliance) |
| Errors | 90 days hot, 1 year cold |
12. Using Centralized Logging
| Stack | Components |
|---|---|
| ELK | Elasticsearch + Logstash + Kibana |
| EFK | Fluentd instead of Logstash |
| Loki | Lightweight, label-based |
| Cloud-native | CloudWatch, Stackdriver, Azure Monitor |