Working with Metrics and Monitoring
1. Understanding Metric Types
| Type | Behavior | Example |
|---|---|---|
| Counter | Monotonic up | requests_total |
| Gauge | Up/down value | queue_depth |
| Histogram | Bucketed distribution | request_duration_seconds |
| Summary | Pre-computed quantiles | response_size |
| Meter (Dropwizard) | Rate over time | events/sec |
2. Implementing Application Metrics
Example: Micrometer + Prometheus
MeterRegistry reg = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
Counter created = Counter.builder("orders.created")
.tag("region", "us-east-1")
.register(reg);
Timer timer = Timer.builder("orders.create.duration")
.publishPercentileHistogram()
.register(reg);
timer.record(() -> { createOrder(req); created.increment(); });3. Implementing Infrastructure Metrics
| Source | Tool |
|---|---|
| Host (CPU, mem, disk) | node_exporter |
| Container | cAdvisor, kubelet |
| K8s objects | kube-state-metrics |
| Network | blackbox_exporter, eBPF |
| DB | postgres_exporter, mysqld_exporter |
4. Implementing Business Metrics
| Metric | Example |
|---|---|
| Conversion rate | signups / visitors |
| Revenue / sec | Sum of completed payments |
| Active users | DAU, MAU |
| Funnel stages | Cart → Checkout → Paid |
5. Understanding RED Method (rate, errors, duration)
| Metric | Definition |
|---|---|
| Rate | Requests/sec |
| Errors | % failures |
| Duration | Latency distribution (p50, p99) |
| Best for | Request-driven services |
6. Understanding USE Method (utilization, saturation, errors)
| Metric | Definition |
|---|---|
| Utilization | % time resource busy |
| Saturation | Queue length / wait time |
| Errors | Count of error events |
| Best for | Resources (CPU, disk, network) |
7. Implementing Metric Aggregation
| Mechanism | Detail |
|---|---|
| Pull (Prometheus) | Scrape /metrics endpoint |
| Push (StatsD, OTLP push) | Client emits to collector |
| Federation | Hierarchical Prom servers |
| Long-term store | Thanos, Mimir, Cortex, VictoriaMetrics |
8. Implementing Alerting Rules
Example: Prometheus alert rule
groups:
- name: api-slo
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.01
for: 10m
labels: { severity: page }
annotations:
summary: "5xx error rate > 1% for 10m"
runbook: "https://wiki/runbooks/api-errors"| Pattern | Detail |
|---|---|
| Multi-window multi-burn | SLO burn-rate alerts (Google SRE) |
| Symptom-based | Alert on user impact, not causes |
| Actionable only | Each page must have a clear action |
9. Implementing Dashboards and Visualization
| Tool | Detail |
|---|---|
| Grafana | Multi-source dashboards |
| Datadog / NR | SaaS |
| Kibana | ES-native |
| Best practice | Service overview + per-route + dependency views |
10. Understanding Metric Cardinality
| Cause | Mitigation |
|---|---|
| High-cardinality labels (user id, request id) | Move to traces/logs |
| Per-endpoint labels with regex paths | Normalize: /users/:id not /users/42 |
| Cost | Linear series storage; query slowdown |
| Limit | Prom: keep total series < few million per server |
11. Implementing SLI and SLO Tracking
| SLI Type | Example |
|---|---|
| Availability | good_requests / total_requests |
| Latency | P(latency < threshold) |
| Quality | % complete responses (no degraded) |
| Error budget burn | (1 − SLI) / (1 − SLO) |