Skip to content
Tutorials Center
My learning

Topics

Lesson 113 of 147 ยท Web and Networking

Signals and Graceful Shutdown

Shut Go servers down cleanly - handle SIGINT and SIGTERM with os/signal and signal.NotifyContext, drain requests with http.Server.Shutdown, stop workers and clean up resources.

A server that is stopped abruptly can lose work: a request halfway through a payment, a file half written, a worker that never reports its result. Deployments, autoscaling and Ctrl+C all end a process the same way: the operating system sends it a signal, and a well-behaved program notices the signal, stops accepting new work, finishes what is in progress and then exits. This is called graceful shutdown.

In Go the recipe has three ingredients: os/signal to hear the signal, a context to tell every part of the program that it is time to stop, and http.Server.Shutdown to drain a web server. This lesson puts them together.

What signals are

A signal is a small message the operating system delivers to a process. The ones you meet most often:

Signal Sent when Default effect
SIGINT you press Ctrl+C terminate
SIGTERM kill <pid>, Docker stop, Kubernetes, systemd terminate
SIGHUP the terminal closes (often reused for "reload config") terminate
SIGKILL kill -9 terminate immediately, cannot be caught

Orchestrators like Docker and Kubernetes send SIGTERM, wait a grace period (30 seconds by default in Kubernetes) and then send SIGKILL. So your shutdown has to finish within that time. On Windows only os.Interrupt (Ctrl+C) is generally useful.

sequenceDiagram
    participant O as Orchestrator
    participant P as Go process
    participant R as In-flight requests
    O->>P: SIGTERM
    P->>P: stop accepting new connections
    P->>R: wait for them to finish
    R-->>P: done
    P->>P: close database, flush logs
    P-->>O: exit 0
    Note over O,P: If the grace period ends first,<br/>the orchestrator sends SIGKILL

The shutdown sequence: stop intake, drain work, release resources, exit before the deadline.

Catching a signal with os/signal

signal.Notify(c, sigs...) asks Go to deliver those signals to the channel c instead of killing the program. The channel must be buffered, because signal delivery never blocks and a signal could be dropped if nobody is ready to receive it (go vet warns about an unbuffered channel).

The program below sends itself a SIGINT so it can run on its own, and then reacts like a real service would. Normally the signal would come from outside:

Go Receiving a signal on a channel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

signal.Stop unregisters the channel; after the last registration is removed, the default behavior returns. signal.Ignore makes a signal do nothing at all, and signal.Reset restores the default.

signal.NotifyContext: the modern way

A raw channel works, but your goroutines already understand contexts. signal.NotifyContext (Go 1.16) returns a context that is cancelled when the signal arrives. You pass that context down to workers, servers and database calls, and they all stop through one mechanism.

Go Stopping workers with NotifyContext

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Each worker writes its message into its own slot of a slice, so the output order does not depend on goroutine scheduling. Two details deserve attention:

  • Call stop() as soon as you have noticed the signal. From then on a second Ctrl+C terminates the program immediately, which is what users expect when a shutdown hangs.
  • context.Cause(ctx) (Go 1.20) tells you that the context was cancelled by a signal. See Context for causes and deadlines, and the sync package for WaitGroup.

Graceful shutdown of an HTTP server

http.Server.ListenAndServe blocks forever and returns http.ErrServerClosed after Shutdown is called. Shutdown(ctx) does the careful work: it closes the listeners so no new connections arrive, then waits for active requests to finish, up to the deadline of ctx. If the deadline passes, it returns the context's error and you can call Close to cut the remaining connections.

The standard layout is:

  1. Start ListenAndServe in a goroutine.
  2. Wait for the signal context.
  3. Call Shutdown with a fresh context that has a timeout (the signal context is already cancelled, so it must not be reused here).
Go A server that finishes a slow request before exiting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The slow request, which was already running when the signal arrived, completes normally. Only afterwards does Shutdown return. In a real server you would write srv.ListenAndServe() in the goroutine and treat any error other than http.ErrServerClosed as a real failure. The basics of servers and handlers are in Building HTTP Servers.

Some things Shutdown does not wait for:

  • Hijacked connections, such as WebSockets, are not tracked. Use srv.RegisterOnShutdown(fn) to notify them so they can close themselves.
  • Background goroutines you started from handlers. Track them with a WaitGroup and wait for it after Shutdown returns.
  • Long-lived requests (streaming, long polling). Give handlers a context derived from srv.BaseContext, or from a context you cancel at shutdown, so they can return early.

Order of cleanup

Shut down in the reverse order that you started things, so that nothing uses a resource that was already closed:

  1. Stop accepting new work (the HTTP server, queue consumers).
  2. Wait for in-flight work to finish.
  3. Close dependencies: database pools, caches, message producers.
  4. Flush logs, metrics and traces.
  5. Exit with status 0, or non-zero if shutdown failed.
Go
func run(ctx context.Context) error {
	db, err := openDatabase()
	if err != nil {
		return err
	}
	defer db.Close() // runs after the server has stopped

	srv := newServer(db)
	errCh := make(chan error, 1)
	go func() { errCh <- srv.ListenAndServe() }()

	select {
	case err := <-errCh:
		return err // the server failed to start or crashed
	case <-ctx.Done():
	}

	shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	return srv.Shutdown(shutdownCtx)
}

Keeping this in a function called from main has one more advantage: deferred cleanups run normally, which they would not if you called os.Exit or log.Fatal in the middle (see Logging).

Tips for production

  • Choose a shutdown timeout shorter than the orchestrator's grace period, so you finish before SIGKILL.
  • When the platform uses readiness checks, make your readiness endpoint start failing as soon as the signal arrives, wait a few seconds so the load balancer stops sending traffic, and then call Shutdown.
  • Log when shutdown starts and ends, with the duration, so you can see slow drains.
  • Test the behavior: start the server in a test, trigger shutdown by cancelling a context, and assert that an in-flight request completes (see Testing Concurrent Code).
  • Do not rely on defer in main for work that matters if the process receives SIGKILL: nothing can run then. Make important operations idempotent and recoverable.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which signal can a Go program NOT catch?

    SIGKILL always terminates the process immediately; the operating system does not let programs handle it.
  2. 2. Why must the channel passed to signal.Notify be buffered?

    signal.Notify sends without blocking. With an unbuffered channel and no receiver waiting, the signal would be dropped.
  3. 3. What does http.Server.Shutdown do?

    Shutdown closes the listeners, then waits for in-flight requests. If its context expires first it returns the context error.
  4. 4. Why should you pass a new context with a timeout to Shutdown instead of the signal context?

    NotifyContext cancels its context when the signal arrives. A cancelled context would end the graceful wait before any request could finish.
  5. 5. What error does ListenAndServe return after a successful Shutdown?

    ListenAndServe returns http.ErrServerClosed immediately when Shutdown is called, so treat that value as the normal case.

Summary

  • Orchestrators and terminals stop programs with signals: SIGINT (Ctrl+C) and SIGTERM can be handled, SIGKILL cannot.
  • signal.Notify delivers signals to a buffered channel; signal.NotifyContext turns a signal into context cancellation that every goroutine can observe.
  • Call stop() after the first signal so a second one terminates the process.
  • http.Server.Shutdown(ctx) stops intake and drains active requests; use a fresh context with a timeout, and treat http.ErrServerClosed as success.
  • Track hijacked connections, background goroutines and long-lived requests yourself, with RegisterOnShutdown, a WaitGroup and cancellable contexts.
  • Shut down in reverse start order and finish before the orchestrator's grace period ends.

Next: database/sql: Working with Databases opens the chapter on databases.