Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 39 ยท Interceptors and Security

Logging, Metrics and Recovery Interceptors

Build production interceptors for gRPC in Go: structured logging, request IDs, latency metrics, and panic recovery that returns INTERNAL, with runnable examples.

The three interceptors every production gRPC server ends up with are recovery (one bad request must not crash the process), logging (one line per call with enough detail to debug) and metrics (counters and latency histograms for dashboards and alerts). They are small, they are boring, and they will save you at 3 a.m.

This lesson builds each one. The real grpc-go signatures are shown read-only, and the runnable versions use the same shape with plain Go functions.

grpc-go cannot run on this site's runner, so the runnable code uses a stand-in Handler type. Moving it to real gRPC means changing the function signature to grpc.UnaryServerInterceptor; the bodies stay the same.

Recovery: turn panics into INTERNAL

In Go a panic in one goroutine that nobody recovers kills the whole process. gRPC runs each handler in its own goroutine, so a nil-pointer bug in a rarely used method could take every other call down with it. A recovery interceptor catches the panic, logs the stack and returns INTERNAL to the caller.

Go
func recoveryInterceptor(
    ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler,
) (resp any, err error) {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("panic in %s: %v\n%s", info.FullMethod, r, debug.Stack())
            err = status.Error(codes.Internal, "internal error")
        }
    }()
    return handler(ctx, req)
}

Note the named results (resp any, err error): the deferred function can only change what is returned if the results are named. Do not put the panic value or stack in the message; the client does not need your internals (see status codes).

Go Recovering from a panic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Logging: one structured line per call

Good call logs have the same fields every time so you can filter them: method, status code, duration, request ID, peer address, user (not secrets). Use structured logging (log/slog) so a tool can index the fields.

Go
func loggingInterceptor(
    ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler,
) (any, error) {
    start := time.Now()
    resp, err := handler(ctx, req)
    p, _ := peer.FromContext(ctx)
    slog.Info("rpc",
        "method", info.FullMethod,
        "code", status.Code(err).String(),
        "duration_ms", time.Since(start).Milliseconds(),
        "peer", p.Addr.String(),
    )
    return resp, err
}

Choose the log level by code: OK and client mistakes (INVALID_ARGUMENT, NOT_FOUND) at info or debug, server problems (INTERNAL, UNAVAILABLE, DATA_LOSS) at error. Never log full request bodies for methods that handle passwords, tokens or personal data.

Request IDs

Read x-request-id from incoming metadata or generate one, put it in the context and include it in every log line and in the response header. A user who sends you "request req-7f3a failed" saves you an hour.

Go Structured logging with request IDs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Metrics: counters and latency

For each method track: total calls by status code (a counter), and the duration (a histogram). From these you can compute the "RED" signals: Rate, Errors, Duration. The usual tool is Prometheus with go-grpc-middleware/providers/prometheus, which gives you grpc_server_handled_total{grpc_method,grpc_code} and a latency histogram out of the box:

Go
srvMetrics := grpcprom.NewServerMetrics(
    grpcprom.WithServerHandlingTimeHistogram(),
)
srv := grpc.NewServer(
    grpc.ChainUnaryInterceptor(srvMetrics.UnaryServerInterceptor()),
    grpc.ChainStreamInterceptor(srvMetrics.StreamServerInterceptor()),
)
srvMetrics.InitializeMetrics(srv) // pre-create series so rate() works from the start

Keep label cardinality low: method and code are fine; user IDs or request IDs as labels will blow up the metrics database.

Go A counter and a latency histogram

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Putting them together

Go
srv := grpc.NewServer(
    grpc.ChainUnaryInterceptor(
        recoveryInterceptor, // outermost: catches panics from everything below
        loggingInterceptor,
        metricsInterceptor,
        authInterceptor,     // see the token authentication lesson
    ),
)

Streams need their own versions with grpc.ChainStreamInterceptor. Most libraries ship both.

Common mistakes

  • Logging the request body: leaks personal data and fills disks.
  • Using the error message as a metric label: unbounded cardinality.
  • Recovery placed inside logging: the panic is recovered, but the log line then shows OK because the code was never set. Keep recovery outermost, or make logging aware of the final error.
  • Forgetting streams: a panic in a streaming handler still kills the process.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why must a recovery interceptor use named return values?

    A deferred function can change the return values only if they are named, so it can replace them with an INTERNAL status.
  2. 2. Which status should a recovery interceptor return to the client?

    INTERNAL signals a server bug; details stay in the server log, not in the response.
  3. 3. Which is a bad Prometheus label for gRPC metrics?

    User IDs have unbounded cardinality and overload the metrics store.
  4. 4. What are the RED signals?

    Rate, Errors and Duration summarize the health of a request-driven service.

Summary

  • Recovery turns a panic into INTERNAL so one bad request cannot crash the server; keep it outermost and use named results.
  • Log one structured line per call: method, code, duration, request ID, peer.
  • Count calls by method and code and record latency in a histogram; keep labels low-cardinality.
  • Never log secrets or whole request bodies.
  • Remember to cover both unary and stream interceptors.

Next: TLS and Mutual TLS