Debugging gRPC Applications
1. Enabling Verbose Logging
| Env Var | Effect |
|---|---|
GRPC_GO_LOG_VERBOSITY_LEVEL=99 | Max verbosity (Go) |
GRPC_GO_LOG_SEVERITY_LEVEL=info | Set severity |
GRPC_TRACE=all | C-core tracing |
GRPC_VERBOSITY=DEBUG | C-core verbosity |
2. Using grpc.EnableTracing
Example: Enable internal tracing
import "google.golang.org/grpc"
grpc.EnableTracing = true
// Expose /debug/requests via golang.org/x/net/trace3. Inspecting Wire Traffic with tcpdump
Example: Capture gRPC traffic
sudo tcpdump -i lo -w grpc.pcap port 50051
# Open in Wireshark; decode as HTTP/24. Using Wireshark for gRPC
| Setup | Detail |
|---|---|
| Decode-as | Right-click → Decode as → HTTP/2 |
| Add .proto | Preferences → Protobuf → Add search path |
| TLS keys | Set SSLKEYLOGFILE env to decrypt |
5. Debugging Connection Issues
| Symptom | Check |
|---|---|
| UNAVAILABLE on dial | Server up? Firewall? Wrong port? |
| TLS handshake fail | Cert chain, SNI, ALPN h2 |
| Name resolution | DNS, resolver scheme |
| Use channelz | Inspect subchannel states |
6. Debugging TLS Issues
| Tool | Detail |
|---|---|
openssl s_client -alpn h2 -connect host:443 | Verify TLS + ALPN |
grpcurl -insecure | Quick check ignoring verify |
| Check SAN | Cert must list server hostname in SAN |
7. Debugging Performance Issues
| Tool | Detail |
|---|---|
| pprof | CPU / heap / goroutine profiles |
| channelz | Per-channel/socket metrics |
| Tracing | Spot slow downstream calls |
8. Using pprof for Profiling
Example: Enable pprof
import _ "net/http/pprof"
go http.ListenAndServe("localhost:6060", nil)
// go tool pprof http://localhost:6060/debug/pprof/profile?seconds=309. Debugging Memory Leaks
| Cause | Detail |
|---|---|
| Leaked streams | Always Close/Drain client streams |
| Goroutine leak | Cancel ctx on shutdown paths |
| Large message retention | Avoid holding response refs in caches |
| Tool | go tool pprof -alloc_space |
10. Using gRPC Channelz
Example: Enable channelz
import "google.golang.org/grpc/channelz/service"
service.RegisterChannelzServiceToServer(srv)
// Use grpcdebug: grpcdebug localhost:50051 channelz channels| Info | Detail |
|---|---|
| Channels | Top-level client conns |
| Subchannels | Per-endpoint state |
| Sockets | Bytes sent/received, flow control |
| Servers | Active call counts |