Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 39 ยท Testing and Deployment

Graceful Shutdown, Logging and Observability

Keep a Gin service healthy in production: shut down without dropping requests, structured logs with request IDs using slog, panic recovery, metrics and tracing basics.

Deploying is not the end of the job. Every release restarts your server, and a production service must restart without cutting off the requests it is handling. It must also tell you, in a form machines can search, what it is doing. This lesson covers graceful shutdown with http.Server, structured logging with request IDs, panic recovery, and the basics of metrics and tracing.

Gin and the full deployment environment are not in the in-page runner, so the Gin code is read-only. The runnable programs use net/http on a loopback port to demonstrate shutdown and logging for real. This chapter follows Docker and Deploying a Gin App.

Why graceful shutdown matters

When you deploy, the orchestrator sends SIGTERM (Ctrl+C sends SIGINT), waits a grace period (30 seconds by default in Kubernetes), then sends SIGKILL. If your program simply exits, any request in flight is cut off: users see errors and half-finished writes may be lost. A graceful shutdown does this instead:

  1. Stop accepting new connections (the load balancer first marks the instance not ready).
  2. Let in-flight requests finish, up to a deadline.
  3. Close databases and other resources.
  4. Exit with status 0.

r.Run() cannot do this. You need your own http.Server.

The pattern in Gin

Go
func main() {
	r := SetupRouter()

	srv := &http.Server{
		Addr:              ":8080",
		Handler:           r,
		ReadHeaderTimeout: 5 * time.Second,
	}

	// Ask the OS to cancel ctx on SIGINT or SIGTERM
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	go func() {
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("server error: %v", err)
		}
	}()

	<-ctx.Done() // wait for the signal
	stop()       // a second Ctrl+C now kills the process immediately

	shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
	defer cancel()
	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("forced shutdown: %v", err)
	}
	sqlDB.Close() // then release resources
	log.Println("bye")
}

ListenAndServe returns http.ErrServerClosed as soon as Shutdown is called; that is expected, not an error. Shutdown waits for active requests, but only until its context expires. Pick a timeout a little shorter than the orchestrator's grace period.

See it work

The program below starts a server on a free loopback port, begins a slow request, then calls Shutdown while that request is still running. New connections are refused, but the slow request completes.

Go Shutdown waits for in-flight requests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The slow request was already running, so it finishes with 200 even though shutdown started first. Try changing the shutdown timeout to 50 ms: Shutdown then returns context deadline exceeded, and in real code you would call srv.Close() to cut off whatever is left.

Structured logging with slog

Plain text logs are hard to search. Structured logs emit one JSON object per line with consistent fields, so log tools can filter by status code, user, route or duration. Go's standard library has log/slog for this. Gin's default logger prints text; for production replace it with your own middleware.

A good access-log middleware records the method, path, status, duration, client IP and a request ID. The request ID is the thread that ties together every log line and the response the user saw: return it in an X-Request-ID header so support can ask for it.

Go
func RequestLogger(log *slog.Logger) gin.HandlerFunc {
	return func(c *gin.Context) {
		start := time.Now()

		id := c.GetHeader("X-Request-ID")
		if id == "" {
			id = uuid.NewString()
		}
		c.Set("request_id", id)
		c.Header("X-Request-ID", id)

		c.Next() // run the rest of the chain

		log.LogAttrs(c.Request.Context(), levelFor(c.Writer.Status()), "request",
			slog.String("request_id", id),
			slog.String("method", c.Request.Method),
			slog.String("route", c.FullPath()), // pattern, not the raw URL
			slog.Int("status", c.Writer.Status()),
			slog.Duration("duration", time.Since(start)),
			slog.String("ip", c.ClientIP()),
		)
	}
}

Log c.FullPath() (/users/:id) rather than the real path so that metrics and searches group by route and do not create a unique value per user. Never log request bodies, passwords, tokens or full Authorization headers.

Go A JSON access log with request IDs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Notice that the level follows the status code: 2xx is INFO, 4xx WARN and 5xx ERROR, so alerts can key on ERROR.

Panic recovery

A panic inside a handler would otherwise kill the request goroutine's connection. gin.Recovery() catches it, logs the stack and answers 500. In production use gin.CustomRecovery to log through slog and return your standard JSON error, and include the request ID so the stack trace can be found. See Gin Recovery and Logging for how the built-in pair works.

Go
r.Use(gin.CustomRecovery(func(c *gin.Context, err any) {
	slog.Error("panic", "request_id", c.GetString("request_id"), "error", err)
	c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": "internal error"})
}))

Metrics and tracing

Logs tell you what happened to one request; metrics tell you how the system behaves overall; traces show where time went inside one request across services. The usual stack:

  • Metrics: expose /metrics with the Prometheus client library (promhttp.Handler(), wrapped by gin.WrapH). Track the "RED" signals: Rate (requests per second), Errors (5xx ratio) and Duration (a latency histogram), labelled by route pattern and method. Keep labels low-cardinality: never label by user ID or raw URL.
  • Tracing: OpenTelemetry's otelgin middleware starts a span per request and propagates the trace ID to downstream calls. Add the trace ID to your logs, and one click takes you from a slow request to its trace.
  • Profiling: net/http/pprof on an internal-only port finds CPU and memory hotspots.
  • Alerts: page people on symptoms (error rate, latency, failing readiness), not on every cause.

Protect /metrics and /debug/pprof: they reveal internals. Bind them to a private port or require authentication.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does srv.Shutdown(ctx) do?

    Shutdown is graceful: in-flight requests may finish, but only until the context deadline.
  2. 2. Why is http.ErrServerClosed not treated as a failure?

    After Shutdown or Close, ListenAndServe returns this sentinel to signal the normal stop.
  3. 3. Why log c.FullPath() instead of the raw URL path?

    Route patterns keep logs and metric labels low-cardinality and searchable.
  4. 4. What is the purpose of a request ID?

    Return it in a header so users can quote it, and include it in every log line.
  5. 5. Which is a bad Prometheus label?

    A label with unbounded values creates millions of time series and can overwhelm the metrics system.

Summary

  • Build your own http.Server, listen for SIGTERM with signal.NotifyContext, call Shutdown with a timeout, then close the database.
  • Log JSON with slog: method, route pattern, status, duration and a request ID returned in X-Request-ID.
  • Use custom recovery to log panics with the request ID and answer a generic 500.
  • Expose RED metrics, add tracing, protect internal endpoints, alert on symptoms.

Next: Project: A To-Do REST API