Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 39 ยท Load Balancing and Gateways

gRPC Health Checking and Server Reflection

Use the standard gRPC health checking protocol with Kubernetes and load balancers, and server reflection with grpcurl, with Go examples and a runnable health service simulation.

Two small, standard services turn a gRPC server from a black box into something that operations tools can work with. The health checking protocol lets load balancers, orchestrators and clients ask "are you able to serve requests?". Server reflection lets tools such as grpcurl ask "what methods do you have?" without needing your .proto files. Both are defined by the gRPC project, implemented in every language, and take a few lines to enable.

The real grpc-go packages are not available on this site, so setup code is shown read-only. The health service logic (status per service name, watching for changes) is simulated with runnable Go.

The health checking protocol

The protocol is itself a gRPC service, grpc.health.v1.Health:

PROTO
syntax = "proto3";
package grpc.health.v1;

message HealthCheckRequest {
  string service = 1;   // "" means the server as a whole
}

message HealthCheckResponse {
  enum ServingStatus {
    UNKNOWN = 0;
    SERVING = 1;
    NOT_SERVING = 2;
    SERVICE_UNKNOWN = 3;  // only for Watch
  }
  ServingStatus status = 1;
}

service Health {
  rpc Check(HealthCheckRequest) returns (HealthCheckResponse);
  rpc Watch(HealthCheckRequest) returns (stream HealthCheckResponse);
}

Check answers once. Watch is a server stream: you get the current status immediately and a new message whenever it changes, which is far cheaper than polling. An empty service name asks about the whole server; you can also report the status of each named service separately (for example users.v1.UserService may be NOT_SERVING while its database is down).

Enabling it in Go (real API)

Go
import (
    "google.golang.org/grpc/health"
    healthpb "google.golang.org/grpc/health/grpc_health_v1"
)

srv := grpc.NewServer()
healthSrv := health.NewServer()          // starts as SERVING for ""
healthpb.RegisterHealthServer(srv, healthSrv)
pb.RegisterUserServiceServer(srv, &userServer{})

// while starting up or when a dependency is down:
healthSrv.SetServingStatus("users.v1.UserService", healthpb.HealthCheckResponse_NOT_SERVING)

// on shutdown, flip first and then drain:
healthSrv.Shutdown()   // marks every service NOT_SERVING
time.Sleep(5 * time.Second)
srv.GracefulStop()

On the client side you can ask the same question, or ask the balancer to use health checking by adding "healthCheckConfig": {"serviceName": "users.v1.UserService"} to the service config (together with importing google.golang.org/grpc/health), so unhealthy backends are skipped. See load balancing.

Running probes in Kubernetes

Kubernetes (1.24+) supports gRPC probes natively. No helper binary needed:

YAML
livenessProbe:
  grpc:
    port: 50051
  initialDelaySeconds: 10
readinessProbe:
  grpc:
    port: 50051
    service: users.v1.UserService
  periodSeconds: 5
  • Liveness asks "is the process stuck?". If it fails, the container is restarted. Keep it cheap and independent of downstream services; a database outage must not cause a restart loop.
  • Readiness asks "should I receive traffic?". It may reflect dependencies. If it fails, the pod is removed from endpoints but not restarted.

For older clusters or other platforms, the grpc_health_probe command-line tool performs the same Check call.

A simulation of Check and Watch

The program implements the service logic with channels: a status map plus watchers that are notified on every change.

Go A health service with Check and Watch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Note the difference: Check on an unknown service is an error (NOT_FOUND), while Watch reports SERVICE_UNKNOWN and keeps the stream open, because the service may be registered later.

A good readiness signal

Deciding what "healthy" means is the real design work. A good readiness check is cheap, reflects whether this instance can do its job, and does not cascade. A sample policy in code:

Go Aggregating dependency checks into one status

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

A missing cache slows things down but does not stop them, so the instance stays in rotation. A missing database means it cannot serve, so it leaves rotation. Never mark every instance NOT_SERVING because one shared downstream is slow; the whole fleet would leave rotation at once.

Server reflection

Reflection is another standard service (grpc.reflection.v1.ServerReflection) that lets a client list services and fetch their descriptors at run time. Tools use it to talk to your server without the .proto files.

Go
import "google.golang.org/grpc/reflection"

reflection.Register(srv)

Then from a terminal:

Terminal
grpcurl -plaintext localhost:50051 list
grpcurl -plaintext localhost:50051 describe users.v1.UserService
grpcurl -plaintext -d '{"id": "42"}' localhost:50051 users.v1.UserService/GetUser

list prints the service names, describe prints a service or message as proto text, and the last call sends a JSON request that grpcurl converts to protobuf using the reflected schema. GUI tools such as Postman, BloomRPC successors and gRPC UI use reflection the same way.

Reflection exposes your whole API surface. That is perfect in development and staging. In production either disable it, or put it behind authentication (for example in an interceptor that allows reflection only for operators), unless the API is public anyway.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an empty service name mean in a HealthCheckRequest?

    The empty string asks about the server as a whole; named services can have their own status.
  2. 2. Which RPC streams status changes instead of polling?

    Watch is a server stream that sends the current status and then every change.
  3. 3. A database outage should make which Kubernetes probe fail?

    Restarting does not fix a database outage. Readiness removes the pod from rotation without a restart loop.
  4. 4. Why should reflection often be disabled or protected in production?

    Reflection lists all services and messages, which helps attackers, so restrict it unless the API is public.

Summary

  • grpc.health.v1.Health offers Check (once) and Watch (stream) per service name; the empty name is the whole server.
  • Enable it with health.NewServer(), set statuses as dependencies change, and flip to NOT_SERVING before graceful shutdown.
  • Kubernetes has native gRPC liveness and readiness probes; keep liveness independent of dependencies.
  • Reflection (reflection.Register) lets grpcurl and GUI tools discover and call methods without .proto files.
  • Protect or disable reflection in production when the API is not public.

Next: REST Transcoding with gRPC-Gateway