Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 39 ยท Services and RPC Types

gRPC Deadlines, Timeouts and Cancellation

Learn why every gRPC call needs a deadline, how deadlines propagate across services, how servers detect cancellation, and how to handle DEADLINE_EXCEEDED, with runnable Go stand-ins.

A call that never finishes is worse than a call that fails. It holds a connection, a goroutine, memory, and a user's patience. gRPC solves this with deadlines: an absolute point in time by which the call must complete. When the deadline passes, the client stops waiting, the server is told, and the call ends with the status DEADLINE_EXCEEDED.

This lesson explains deadlines versus timeouts, how they travel through a chain of services, and how to write handlers that stop working when a call is cancelled. The real gRPC library is read-only here; the Go stand-ins use the real context package, which is exactly what gRPC-Go uses.

Timeout versus deadline

  • A timeout is a duration: "give this call two seconds".
  • A deadline is a moment: "this call must finish by 12:00:02".

Clients usually think in timeouts, and the library converts them into a deadline. On the wire the deadline travels as the grpc-timeout header (a remaining duration like 1500m, meaning 1500 milliseconds), which the next hop converts back into its own deadline. See how gRPC uses HTTP/2.

Setting a deadline (read-only)

Go
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

resp, err := client.GetProduct(ctx, &pb.GetProductRequest{Id: 7})
if status.Code(err) == codes.DeadlineExceeded {
	// the server was too slow, or unreachable
}

Never omit the deadline in production code. The default for gRPC is no deadline at all, so a hung server can block callers forever. Python uses timeout=2.0, Java uses stub.withDeadlineAfter(2, SECONDS).

defer cancel() is required even when the call finishes early; it releases the timer and resources associated with the context.

What a server sees

When the deadline expires, or the client cancels, the server's context is cancelled. Well-behaved handlers check it:

Go
func (s *server) Search(ctx context.Context, req *pb.SearchRequest) (*pb.SearchResponse, error) {
	for _, shard := range s.shards {
		if err := ctx.Err(); err != nil {
			return nil, status.FromContextError(err).Err() // DEADLINE_EXCEEDED or CANCELED
		}
		// ... query shard, passing ctx to the database client as well ...
	}
	return resp, nil
}

ctx.Err() returns context.DeadlineExceeded or context.Canceled. The helper status.FromContextError maps them to DEADLINE_EXCEEDED and CANCELED. Pass ctx to every downstream call (database, HTTP client, other gRPC services), so that cancellation reaches the whole chain.

If a handler ignores the context, the server keeps burning CPU for a result nobody is waiting for. At scale this wasted work amplifies overload.

Stand-in 1: a handler that stops when cancelled

The handler pretends to do five steps. With a generous deadline all five finish. With a 100 ms deadline and 300 ms steps, the handler notices the expired context in the middle of the first step and stops, reporting how far it got. With a context cancelled upfront it does nothing at all.

Go Stand-in: honoring the context

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Deadline propagation across services

In a microservice chain, A calls B and B calls C. If A gave the request 2 seconds, then B and C must finish inside that same budget. gRPC does this automatically when you pass the incoming context to outgoing calls: the outgoing call inherits the remaining time. If B creates a fresh context.Background() instead, the budget is lost and C might keep working long after A gave up.

sequenceDiagram
  participant A as Service A
  participant B as Service B
  participant C as Service C
  A->>B: deadline in 2000 ms
  Note over B: uses 300 ms
  B->>C: deadline in about 1700 ms
  Note over C: uses 1800 ms
  C--xB: DEADLINE_EXCEEDED
  B--xA: DEADLINE_EXCEEDED

The deadline shrinks as time is consumed, and the failure travels back up the chain.

Stand-in 2: propagating a budget through hops

The chain below passes one ctx down. Each hop checks whether a deadline exists and that the time left never grows. We print booleans rather than exact times so the output is stable.

Go Stand-in: the budget shrinks at every hop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Choosing deadline values

  • Start from the user experience: a web request that must answer in 1 second leaves less than that for backend calls.
  • Pick per-method values. A GetProduct might need 200 ms; a report export might need 30 s. Use the service config from clients, channels and stubs for defaults.
  • Leave headroom for retries; they share the single deadline.
  • Do not set a deadline shorter than typical latency percentiles (p99), or healthy calls will fail.
  • Long-lived streams can use a deadline for the whole stream, or application-level timeouts for each message.

Cancellation by the client

Calling cancel() on the client context aborts the call: the client sends an RST_STREAM frame, the server's context is cancelled, and the client receives CANCELED. Typical triggers are a user closing a page or another call in a fan-out finishing first.

Common mistakes

  1. No deadline, so hung calls pile up.
  2. Creating context.Background() in a handler instead of passing the request context.
  3. Ignoring ctx in long loops.
  4. Treating DEADLINE_EXCEEDED as a reason to retry immediately; the server may still be working, and retries add load.
  5. Forgetting defer cancel(), which leaks timers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the default deadline of a gRPC call if you set none?

    Without an explicit deadline a call may block indefinitely, which is why you should always set one.
  2. 2. Service B receives a request with 1.5 s left and calls C using the incoming context. How much time does C get?

    The deadline is absolute. Time already spent by B reduces the budget available to C.
  3. 3. Which status does a call get when the client cancels it?

    DEADLINE_EXCEEDED is for expiry; CANCELED is for explicit cancellation.
  4. 4. Why should server handlers check ctx.Err() in long loops?

    Cancelled calls keep running unless the handler notices the context.

Summary

  • A deadline is an absolute time; clients usually set it as a timeout.
  • The deadline travels as grpc-timeout and shrinks across hops when you pass the context along.
  • Servers must watch the context and stop when it is done; map errors with status.FromContextError.
  • Always set deadlines, and choose them per method with headroom.
  • Always defer cancel().

Next: Server Streaming