Implementing Metrics and Monitoring
1. Using Prometheus Client
Example: Register collector + endpoint
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
http.Handle("/metrics", promhttp.Handler())
go http.ListenAndServe(":9090", nil)
| Library | Detail |
|---|---|
client_golang | Official Go client |
go-grpc-middleware/providers/prometheus | Drop-in gRPC interceptor |
otelgrpc + OTLP metrics | Modern alternative |
2. Creating Counter Metrics
Example: Counter
reqs := prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "grpc_requests_total"},
[]string{"method", "code"},
)
prometheus.MustRegister(reqs)
reqs.WithLabelValues(method, code).Inc()
3. Creating Histogram Metrics
Example: Histogram for latency
lat := prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "grpc_request_duration_seconds",
Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2, 5},
},
[]string{"method"},
)
lat.WithLabelValues(method).Observe(dur.Seconds())
4. Creating Gauge Metrics
| Use | Detail |
|---|---|
| Active streams | gauge.Inc/Dec on open/close |
| Queue depth | Sample current value |
| Connection count | Track via state callbacks |
5. Registering Metrics
| API | Detail |
|---|---|
prometheus.MustRegister | Global default registry |
| Custom registry | prometheus.NewRegistry() for test isolation |
6. Exposing Metrics Endpoint
| Endpoint | Path |
|---|---|
| Prometheus scrape | /metrics |
| Separate port | :9090 common |
| Auth | Internal network or basic auth |
7. Instrumenting with Interceptor
Example: Metric interceptor
func Metrics(reqs *prometheus.CounterVec, lat *prometheus.HistogramVec) grpc.UnaryServerInterceptor {
return func(ctx context.Context, req any, info *grpc.UnaryServerInfo,
h grpc.UnaryHandler) (any, error) {
start := time.Now()
res, err := h(ctx, req)
code := status.Code(err).String()
reqs.WithLabelValues(info.FullMethod, code).Inc()
lat.WithLabelValues(info.FullMethod).Observe(time.Since(start).Seconds())
return res, err
}
}
8. Tracking Request Count
| Labels | Detail |
|---|---|
| method | Full RPC name |
| code | Status code |
| Avoid | User id / request id (unbounded) |
9. Tracking Request Duration
| Metric Type | Trade-off |
|---|---|
| Histogram | Bucketed, fixed memory, accurate quantiles via PromQL |
| Summary | Client-side quantiles; can't aggregate across pods |
| Native histograms Prom 2.40+ | Higher resolution |
10. Adding Custom Labels
| Rule | Detail |
|---|---|
| Bounded cardinality | < ~100 unique values per label |
| Static at registration | Cannot add labels after creation |
11. Monitoring Error Rates
Example: PromQL error rate
sum(rate(grpc_requests_total{code!="OK"}[5m])) by (method)
/
sum(rate(grpc_requests_total[5m])) by (method)
12. Tracking Active Streams
| Pattern | Detail |
|---|---|
| Stream interceptor | activeStreams.Inc() on entry, Dec() on return |
| By type | Label with server/client streaming |