Implementing Metrics and Monitoring

1. Using Prometheus Client

Example: Register collector + endpoint

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)
http.Handle("/metrics", promhttp.Handler())
go http.ListenAndServe(":9090", nil)
LibraryDetail
client_golangOfficial Go client
go-grpc-middleware/providers/prometheusDrop-in gRPC interceptor
otelgrpc + OTLP metricsModern alternative

2. Creating Counter Metrics

Example: Counter

reqs := prometheus.NewCounterVec(
    prometheus.CounterOpts{Name: "grpc_requests_total"},
    []string{"method", "code"},
)
prometheus.MustRegister(reqs)
reqs.WithLabelValues(method, code).Inc()

3. Creating Histogram Metrics

Example: Histogram for latency

lat := prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "grpc_request_duration_seconds",
        Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2, 5},
    },
    []string{"method"},
)
lat.WithLabelValues(method).Observe(dur.Seconds())

4. Creating Gauge Metrics

UseDetail
Active streamsgauge.Inc/Dec on open/close
Queue depthSample current value
Connection countTrack via state callbacks

5. Registering Metrics

APIDetail
prometheus.MustRegisterGlobal default registry
Custom registryprometheus.NewRegistry() for test isolation

6. Exposing Metrics Endpoint

EndpointPath
Prometheus scrape/metrics
Separate port:9090 common
AuthInternal network or basic auth

7. Instrumenting with Interceptor

Example: Metric interceptor

func Metrics(reqs *prometheus.CounterVec, lat *prometheus.HistogramVec) grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req any, info *grpc.UnaryServerInfo,
        h grpc.UnaryHandler) (any, error) {
        start := time.Now()
        res, err := h(ctx, req)
        code := status.Code(err).String()
        reqs.WithLabelValues(info.FullMethod, code).Inc()
        lat.WithLabelValues(info.FullMethod).Observe(time.Since(start).Seconds())
        return res, err
    }
}

8. Tracking Request Count

LabelsDetail
methodFull RPC name
codeStatus code
AvoidUser id / request id (unbounded)

9. Tracking Request Duration

Metric TypeTrade-off
HistogramBucketed, fixed memory, accurate quantiles via PromQL
SummaryClient-side quantiles; can't aggregate across pods
Native histograms Prom 2.40+Higher resolution

10. Adding Custom Labels

RuleDetail
Bounded cardinality< ~100 unique values per label
Static at registrationCannot add labels after creation

11. Monitoring Error Rates

Example: PromQL error rate

sum(rate(grpc_requests_total{code!="OK"}[5m])) by (method)
/
sum(rate(grpc_requests_total[5m])) by (method)

12. Tracking Active Streams

PatternDetail
Stream interceptoractiveStreams.Inc() on entry, Dec() on return
By typeLabel with server/client streaming