Monitoring and Logging
1. Implementing Structured Logging
| Library | Lang |
|---|
| pino | Node (fast JSON) |
| winston | Node (flexible) |
| zap | Go |
| logback | Java |
| structlog | Python |
Example: pino structured
import pino from "pino";
const log = pino({ level: process.env.LOG_LEVEL ?? "info" });
log.info({ event:"ws_open", userId, ip, ua, sockId }, "connection opened");
2. Logging Connection Events
| Event | Fields |
|---|
| ws_open | sockId, userId, ip, ua, ts |
| ws_close | sockId, code, reason, duration, wasClean |
| ws_error | sockId, err message |
| ws_auth_ok / fail | sockId, userId, reason |
3. Logging Message Events
Warning: Never log raw message payloads at info level — PII risk. Log size, type, and id only.
| Field | Detail |
|---|
| type | Message type |
| id | Correlation |
| size | Bytes |
| dir | in / out |
| latency_ms | For RPCs |
4. Tracking Connection Metrics
| Metric | Type |
|---|
ws_connections_active | Gauge |
ws_connections_opened_total | Counter |
ws_connections_closed_total | Counter (by code) |
ws_connection_duration_seconds | Histogram |
5. Monitoring Message Throughput
| Metric | Detail |
|---|
| msgs_sent_total / sec | Outbound rate |
| msgs_received_total / sec | Inbound rate |
| bytes_sent / received | Bandwidth |
| buffer_high_watermark | Backpressure signal |
6. Tracking Error Rates
| Metric | Tag By |
|---|
| parse_errors_total | schema |
| auth_errors_total | reason |
| rate_limited_total | tier |
| abnormal_close_total | code |
| Tool | Strength |
|---|
| Datadog APM | Distributed traces + logs |
| New Relic | Full-stack |
| Sentry | Error tracking + traces |
| OpenTelemetry | Vendor-neutral |
8. Implementing Custom Metrics
Example: prom-client
import client from "prom-client";
const active = new client.Gauge({ name:"ws_connections_active", help:"" });
wss.on("connection", (ws) => {
active.inc();
ws.on("close", () => active.dec());
});
9. Setting Up Alerts
| Alert | Trigger |
|---|
| Connection drop spike | close_rate > 3× baseline |
| Memory near cap | RSS > 85% |
| Auth failure surge | auth_errors / sec spike |
| Pub/sub lag | End-to-end > 1s |
10. Analyzing Logs
| Tool | Detail |
|---|
| Loki + Grafana | Lightweight, label-based |
| ELK | Elasticsearch + Kibana |
| Splunk | Enterprise |
| Datadog Logs | Unified with metrics/traces |