Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 39 ยท Error Handling and Metadata

gRPC Deadlines and Cancellation

Learn why every gRPC call needs a deadline, how deadlines propagate across services, and how to honour cancellation in Go handlers, with runnable context examples.

A call without a time limit is a promise to wait forever. If a downstream service hangs, your service hangs; the callers of your service hang; threads, goroutines and memory pile up until something falls over. gRPC's answer is the deadline: an absolute point in time by which the call must be finished. When the deadline passes, the call ends with DEADLINE_EXCEEDED.

gRPC also has cancellation: the client can abandon a call at any time, and the server is told. Both are built on the same idea, and in Go both are exposed through context.Context.

The real gRPC library cannot run on this site. The go blocks outside a TryIt show the real API. The runnable blocks use only the standard context package, which is exactly what grpc-go uses, so the behaviour you see is the real behaviour of deadlines and cancellation.

Deadline or timeout?

A timeout is a duration ("5 seconds from now"). A deadline is a point in time ("12:00:05.000"). The client API accepts either, but on the wire gRPC sends the remaining time in the grpc-timeout header, and each hop converts it back to a deadline on its own clock. This avoids trouble with clocks that disagree between machines.

By default a gRPC call has no deadline. Always set one.

Setting a deadline on the client (real API)

Go
ctx, cancel := context.WithTimeout(context.Background(), 800*time.Millisecond)
defer cancel() // always call cancel to free resources

resp, err := client.GetUser(ctx, &pb.GetUserRequest{Id: "42"})
if err != nil {
    if status.Code(err) == codes.DeadlineExceeded {
        log.Println("too slow")
    }
    return err
}

defer cancel() is not optional: it releases the timer even when the call returns early.

What the server sees

The deadline arrives as the handler's ctx. When the time runs out (or the client cancels), ctx.Done() closes and ctx.Err() is context.DeadlineExceeded or context.Canceled. A well-behaved handler checks the context during long work and stops:

Go
func (s *server) Report(ctx context.Context, req *pb.ReportRequest) (*pb.Report, error) {
    for _, shard := range s.shards {
        if err := ctx.Err(); err != nil {
            return nil, err // grpc-go maps this to DEADLINE_EXCEEDED / CANCELLED
        }
        // ... query shard
    }
    return &pb.Report{}, nil
}

Even if your handler ignores the context, grpc-go has already answered the client with DEADLINE_EXCEEDED. The difference is that the server keeps burning CPU for a response nobody will read.

Runnable: a deadline in action

Go A slow worker meets a deadline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Deadline propagation

The most important rule for microservices: pass the incoming ctx to every outgoing call. The remaining time then shrinks as the request travels, so a chain A to B to C never lasts longer than A's original deadline.

flowchart LR
    U["User<br/>deadline 1000 ms"] --> A["Service A<br/>spent 200 ms"]
    A -->|"remaining 800 ms"| B["Service B<br/>spent 300 ms"]
    B -->|"remaining 500 ms"| C["Service C"]

Each hop forwards what is left. If C is slow, everyone gives up at the same moment.

Go Propagating a deadline down a chain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Run it: C starts with roughly 250 ms left but needs 400 ms, so the whole chain fails together instead of C working for a caller who already left.

A server can also tighten the deadline for its own downstream calls (context.WithTimeout(ctx, ...) creates a shorter child) but can never extend the caller's deadline.

Cancellation

A client cancels by calling the cancel function of its context. The server's ctx.Done() closes within a short moment and the status is CANCELLED. The same happens automatically when a client process dies or a connection breaks. For streaming calls, cancelling the context ends the stream on both sides.

Go Cancelling a call

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Choosing deadlines

  • Base them on how long a user is willing to wait, then divide the budget among the hops.
  • Set deadlines at the edge (the first service the user's request touches); inner services inherit them.
  • Short for cheap lookups (tens to hundreds of milliseconds), longer for batch work. Streams that live for hours need per-message timeouts in your own code, since the stream deadline covers the whole call.
  • Leave some slack: a retry budget (see retries and hedging) must fit inside the deadline.

Common mistakes

  • Forgetting defer cancel(): it leaks timers.
  • Using context.Background() inside a handler for downstream calls: the chain loses the deadline and the cancellation.
  • Treating DEADLINE_EXCEEDED as proof that nothing happened. The server may have finished the work after the client gave up, so writes need idempotency.
  • Writing very long deadlines "to be safe". It only moves the failure later.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the default deadline of a gRPC call?

    Without a deadline a call may wait indefinitely, which is why every client call should set one.
  2. 2. What should a handler pass to its own outgoing gRPC calls?

    Passing the incoming context forwards the remaining time and also propagates cancellation.
  3. 3. Which status does a client see when it cancels its own call?

    Cancellation produces CANCELLED, whereas an expired deadline produces DEADLINE_EXCEEDED.
  4. 4. Why does a DEADLINE_EXCEEDED on a write call not prove the write failed?

    The client only knows it stopped waiting. The server might have finished, so retries of writes need idempotency.

Summary

  • A deadline is the absolute time by which a call must finish; always set one on the client.
  • gRPC sends the remaining time (grpc-timeout), and each hop turns it into its own deadline.
  • Handlers receive the deadline and cancellation through ctx; check ctx.Err() or ctx.Done() in long work.
  • Pass the incoming context to downstream calls so the budget shrinks along the chain.
  • Always defer cancel(), and make writes idempotent because a timed-out call may still have succeeded.

Next: Retries and Hedging