Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 39 ยท Projects and Practice

A gRPC Production Checklist

A practical checklist for running gRPC services in production: API design, deadlines, retries, security, observability, shutdown and deployment, with runnable shutdown and config-audit examples.

A gRPC service that works on your laptop is a prototype. A service that survives deploys, traffic spikes, slow dependencies, bad clients and 3 a.m. incidents is production ready. The difference is mostly a list of small, unglamorous decisions. This lesson collects them into one checklist you can use for new services and for reviews of old ones, with two runnable examples for the trickiest items: graceful shutdown and automatic configuration audits.

The gRPC library cannot run on this site, so snippets that use it are read-only. The runnable programs simulate the shutdown sequence and a configuration audit in plain Go.

1. API design

  • Versioned package (orders.v1), one request and one response message per RPC.
  • UNSPECIFIED zero values for enums; field numbers never reused; removed fields reserved (schema evolution).
  • Pagination on every list RPC (page_size, page_token, next_page_token) from day one.
  • Idempotency keys on create operations that clients may retry.
  • buf lint and buf breaking run in CI (Buf).
  • Documentation comments in the .proto; they become generated docs.
  • Field-level limits documented (string lengths, list sizes) and enforced on the server.

2. Timeouts, retries and resilience

  • Every client call has a deadline; deadlines propagate through handlers (deadlines).
  • Handlers stop work when ctx is cancelled.
  • Retries only for idempotent calls and UNAVAILABLE; backoff with jitter; throttling configured (retries).
  • Concurrency limits or load shedding: return RESOURCE_EXHAUSTED (or UNAVAILABLE) early instead of queuing forever.
  • Circuit breaking or outlier detection for slow dependencies.
  • Max message sizes set deliberately; large payloads streamed or stored elsewhere.

3. Security

  • TLS everywhere outside localhost; mTLS between services where possible (TLS).
  • Authentication interceptor with deny-by-default authorization (tokens).
  • Reflection disabled or protected in production.
  • Input validation on every field; size limits on strings, lists and metadata.
  • No secrets, tokens or personal data in logs, error messages or metadata labels.
  • Dependencies and protobuf libraries kept up to date.

4. Observability

  • Recovery, logging and metrics interceptors for unary and stream calls (interceptors).
  • Request IDs in every log line and in response headers; trace context propagated (OpenTelemetry gRPC instrumentation).
  • Dashboards for rate, errors by status code and p50/p99 latency per method; alerts on error budget burn rather than single errors.
  • Health service with liveness and readiness probes (health checks).
  • Profiling (pprof) available on an internal port.

5. Deployment

  • Graceful shutdown on SIGTERM (next section).
  • Load balancing verified under load with per-backend counters (load balancing); connections recycled with MaxConnectionAge.
  • Keepalive configured consistently on both sides (tuning).
  • Resource requests and limits sized from a load test; GOMAXPROCS/GOMEMLIMIT matched to container limits.
  • Rolling deploys tested while a load generator runs: zero errors expected.
  • Rollback plan, and backward-compatible schema changes only (old and new versions run side by side during a rollout).

Graceful shutdown

Deploys, autoscaling and node drains all stop your process. A good shutdown sequence avoids errors for in-flight and incoming calls:

  1. Receive SIGTERM.
  2. Mark the health status NOT_SERVING so load balancers and readiness probes stop sending new calls.
  3. Wait a few seconds so the change propagates (endpoints update, clients re-resolve).
  4. Call GracefulStop(): no new calls, in-flight calls finish.
  5. If it takes longer than a limit (streams can live forever), call Stop() to cut the rest.
Go
func run(srv *grpc.Server, healthSrv *health.Server, lis net.Listener) error {
    errCh := make(chan error, 1)
    go func() { errCh <- srv.Serve(lis) }()

    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
    defer stop()

    select {
    case err := <-errCh:
        return err
    case <-ctx.Done():
    }

    healthSrv.Shutdown()                 // all services NOT_SERVING
    time.Sleep(5 * time.Second)          // let balancers notice

    done := make(chan struct{})
    go func() { srv.GracefulStop(); close(done) }()
    select {
    case <-done:
    case <-time.After(25 * time.Second): // below Kubernetes terminationGracePeriodSeconds (30 s)
        srv.Stop()
    }
    return nil
}

The sequence below simulates it with goroutines standing in for in-flight calls. Slow calls are given a deadline; the ones that overrun are cut.

Go Graceful shutdown: drain, wait, then force

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Notice who finishes cleanly (the short calls), who is cut off (the stream that would have run for five seconds) and that new calls are rejected once stopping begins.

Auditing your configuration automatically

Checklists drift. A small test or startup check can catch the most important mistakes: no TLS, no max connection age, reflection on in production, missing deadlines. This program audits a config struct against a few rules, the kind of check you could run in CI against rendered Helm values or environment variables.

Go A configuration audit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Operations habits

  • Runbooks: for each alert, a short document that says what it means and what to do. Include the common gRPC culprits: UNAVAILABLE spikes (deploys, connection resets), DEADLINE_EXCEEDED (slow downstream), RESOURCE_EXHAUSTED (limits, message size).
  • Game days: kill a backend, add latency, expire a certificate in staging, and see what the system does.
  • Capacity: know your per-instance limit from a load test and keep headroom for losing one zone.
  • Certificates: monitor expiry dates and automate renewal.
  • Client libraries: publish generated clients as versioned packages with sensible defaults (deadlines, retry policy) baked in, so every team does not reinvent them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What should happen first when a server receives SIGTERM?

    Flipping health first lets balancers stop routing new calls before in-flight ones are drained.
  2. 2. Why use GracefulStop with a time limit and Stop afterwards?

    GracefulStop waits for all calls, including streams. A limit guarantees the process exits within the grace period.
  3. 3. Which item makes clients rebalance across new backends?

    Recycling connections makes clients reconnect and re-resolve, which spreads load across current backends.
  4. 4. Why should liveness probes not depend on the database?

    Restarting the service does not fix the database; use readiness to stop routing traffic instead.

Summary

  • Production readiness is a checklist across API design, deadlines and retries, security, observability and deployment.
  • Version and validate APIs, enforce them with Buf in CI, and make clients retry safely.
  • Secure with TLS or mTLS, authentication and deny-by-default authorization; protect reflection.
  • Instrument every call with recovery, logging, metrics and tracing; expose health probes.
  • Shut down gracefully: NOT_SERVING, wait, GracefulStop with a limit, then Stop.
  • Automate audits so the checklist does not drift.

Next: Practice: gRPC and Protobuf Exercises