Implementing Monitoring and Metrics
1. Collecting Application Metrics
| Type | Detail |
|---|---|
| Counter | Monotonic; e.g. http_requests_total |
| Gauge | Up/down; e.g. queue_depth |
| Histogram | Latency buckets |
| Summary | Quantiles client-side |
| Library | Micrometer (Java), OpenTelemetry Metrics |
2. Implementing Health Endpoints
| Endpoint | Use |
|---|---|
| /livez | Process alive (restart if fails) |
| /readyz | Ready to serve (remove from LB) |
| /startupz | Slow init complete |
| Spec | Spring Actuator /actuator/health |
3. Using RED Metrics
| Metric | Detail |
|---|---|
| Rate | Requests/sec |
| Errors | Failed req/sec |
| Duration | Latency distribution |
| For | Request-driven services |
4. Using USE Metrics
| Metric | Detail |
|---|---|
| Utilization | % busy time |
| Saturation | Queue length / wait |
| Errors | Error events |
| For | Resources (CPU, disk, net, pool) |
5. Monitoring Business Metrics
| Example | Why |
|---|---|
| Orders/min | Detects revenue drop |
| Signups/hour | Marketing impact |
| Cart abandonment | UX issue |
| $ per service | Value delivery |
6. Implementing Custom Metrics
Example: Micrometer (Java)
Counter ordersPlaced = Counter.builder("orders.placed")
.tag("region", region)
.register(meterRegistry);
ordersPlaced.increment();
Timer.Sample s = Timer.start(meterRegistry);
processOrder(...);
s.stop(meterRegistry.timer("orders.process.duration"));7. Using Metric Labels
| Rule | Detail |
|---|---|
| Low cardinality | Bounded set (method, status, route) |
| Avoid | userId, requestId as labels |
| Prom limit | Each label combo = new series ($) |
8. Setting Up Alerting Rules
Example: Prometheus alert rule
groups:
- name: orders
rules:
- alert: HighErrorRate
expr: sum(rate(http_requests_total{job="orders",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="orders"}[5m])) > 0.05
for: 10m
labels: { severity: page }
annotations:
summary: "orders error rate > 5% for 10m"| Practice | Detail |
|---|---|
| Symptom-based | Alert on user impact, not causes |
| For: window | Avoid flapping |
| Severity | page / ticket / log |
9. Implementing SLO Monitoring
| Term | Detail |
|---|---|
| SLI | Indicator (e.g. p99 latency) |
| SLO | Target (99.9% < 300ms) |
| Error budget | 1 − SLO; controls release pace |
| Burn rate alerts | 2% of budget in 1h = page |
10. Creating Dashboards
| Layer | Content |
|---|---|
| Overview | RED + SLO status |
| Service detail | Per-endpoint, per-version |
| Resource | USE for CPU/mem/IO |
| Tool | Grafana, Datadog, Kibana, Chronograf |
11. Implementing Metric Aggregation
| Method | Detail |
|---|---|
| Pull | Prometheus scrapes /metrics |
| Push | StatsD / OTLP gateway |
| Recording rules | Pre-aggregate expensive queries |
| Federation | Hierarchical Prom |
12. Using Monitoring Tools
| Tool | Notes |
|---|---|
| Prometheus + Grafana | OSS standard |
| Mimir / Thanos / Cortex | Long-term + HA Prom |
| VictoriaMetrics | Fast TSDB |
| Datadog / New Relic / Dynatrace | Managed APM |
| CloudWatch / GCP Ops | Cloud-native |