Skip to content
Tutorials Center
My learning

Topics

Lesson 102 of 147 ยท Testing and Tooling

Profiling and Debugging

Find slow and broken Go code with printf debugging, panic traces, runtime.Stack, pprof CPU and heap profiles, runtime/trace, the Delve debugger and GODEBUG settings.

Sooner or later a program does something you didn't expect: it returns the wrong answer, crashes with a wall of text, hangs, or runs ten times slower than it should. Go ships with unusually good tools for each of those situations, and most of them are part of the standard library or the go command, so there is nothing extra to install. This lesson walks through them from the cheapest to the most powerful.

You will print values the useful way, read a panic trace line by line, look at goroutines from inside a running program, take CPU, heap and goroutine profiles with runtime/pprof, record an execution trace, step through code in the Delve debugger, and switch runtime behaviors with GODEBUG. Where a tool needs your terminal (go tool pprof, dlv), the session is shown read-only; everything that can run on this site does.

Timings and memory addresses differ on every machine and every run, so the runnable examples print only facts that stay the same: function names, line numbers, counts and yes/no checks. The profile files themselves are real; you just can't open them in a browser here.

Printf debugging, done well

Printing a value is the oldest debugger, and in Go it works well if you pick the right verb (printing with fmt lists them all). %v is the plain form, %+v adds struct field names, %#v prints Go syntax that you could paste back into code, and %T shows the type. For debugging output, %q is better than %s for strings, because it shows invisible characters and empty strings.

The log package adds a prefix and, with log.Lshortfile, the file and line that printed the message. log/slog (logging) does the same with key-value pairs:

Go Debug printing with fmt, log and slog

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Notice that %+v of a struct with a nil pointer prints Note:<nil>, and that %v of the order's pointer field would print an address, which is rarely what you want. Print *o.Note after checking for nil.

A few habits make printf debugging much less painful:

  • Print where the problem is, not where the symptom is. Add a print at the start of the function that returns the wrong value, then one halfway through. Bisect.
  • Label every value. fmt.Println(x, y) can be mistaken for something else; fmt.Printf("x=%v y=%v\n", x, y) can't.
  • Write to stderr (fmt.Fprintln(os.Stderr, ...) or the log package) so that debug output doesn't end up in a pipe that another program reads.
  • Remove the prints, or turn them into real logging with a level. Leftover fmt.Println("here") lines in a commit are a code smell. go vet won't catch them, but your reviewer will.

Reading a panic trace

A panic that nothing recovers (panic and recover) prints a message and a stack trace, then the program exits with status 2. The trace is the most valuable debugging tool you get for free, and it is easy to read once you know the layout. Run this program, which panics three calls deep:

Go An unrecovered panic (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The output starts with the reason, panic: assignment to entry in nil map, followed by goroutine 1 [running]: and then pairs of lines, one pair per function call, from the innermost call outward:

TEXT
panic: assignment to entry in nil map

goroutine 1 [running]:
main.(*Inventory).Remove(...)
	main.go:10
main.process(0xc000012345, {0xc000012350?, 0x0?, 0x70?})
	main.go:15 +0x50
main.main()
	main.go:22 +0x57

Read it from the top: the first frame is where the panic happened (Remove, line 10), and each frame below it is the caller. A (...) instead of an argument list means the compiler inlined that call; otherwise the arguments are printed as raw hex words, with ? marking values that may be stale. The number after the +0x is a byte offset into the function and is rarely useful. Three things to remember:

  • The message (panic: ...) says what happened: index out of range, nil pointer dereference, nil map write, a failed type assertion, a custom panic("...") or a wrapped error.
  • The first frame in your code is usually the one to look at. Frames inside the runtime package or the standard library are signposts, not the bug.
  • In this example the actual bug is not line 10 but the declaration in main: Inventory was never given a map. The trace tells you where it broke; you still have to decide where it went wrong. Good fix: make the zero value useful, or add a constructor.

A panic in a goroutine you started kills the whole program, and the trace shows goroutine 18 [running]: instead of goroutine 1. Only the goroutine that panicked is shown in detail unless you set GOTRACEBACK=all (see the end of this lesson).

Deadlocks print a trace too

When every goroutine is blocked, the runtime notices that nothing can ever make progress and stops the program with a fatal error, not a panic, so recover can't catch it:

Go A deadlock report (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first line is fatal error: all goroutines are asleep - deadlock!, and the trace says what main is waiting for: goroutine 1 [chan send]:. That state in square brackets (chan send, chan receive, select, sync.Mutex.Lock, semacquire, sleep, IO wait) is the fastest clue when a program hangs. Note that the runtime only detects a deadlock when all goroutines are stuck. A program with an extra goroutine that is blocked forever just hangs silently, which is a goroutine leak (see goroutines and concurrency patterns). The next section shows how to find those.

Asking "who called me?" at runtime

Sometimes you don't want a crash, you want to see the call stack at a specific point. The runtime package gives you the pieces. runtime.Caller(skip) returns the file and line of a caller, runtime.Callers plus runtime.CallersFrames returns a whole stack as structured data (correctly handling inlined functions), and runtime.Stack and runtime/debug.Stack return the textual dump you see in a panic. debug.PrintStack() writes it to stderr.

Go runtime.Callers and goroutine dumps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first part prints the call chain with line numbers: callers is called from level3, then level2, level1, main, and finally two functions from the runtime (runtime.main is what calls your main). The second part is what you would do in a "why is my server stuck?" investigation: dump all goroutines and look at their states. Three goroutines are parked in chan receive and the main goroutine is running, because it is the one taking the snapshot.

Production servers get the same view for free from net/http/pprof (below) at the URL /debug/pprof/goroutine?debug=2, or from sending SIGQUIT to the process (Ctrl+\ in a terminal), which makes the runtime dump all goroutine stacks and exit.

Profiling with pprof

A profiler answers "where does the time (or memory) go?" by sampling the running program. Go's profiler is called pprof. The runtime/pprof package writes profile files; the go tool pprof command reads them. The kinds of profile you will use:

Profile Question it answers
cpu Which functions are running on the CPU? (sampled 100 times per second)
heap Which code allocated the memory that is live now? (inuse_space), or allocated the most in total (alloc_space)
allocs The same data as heap, with the sample index set to allocations
goroutine How many goroutines exist, and what are they doing?
block Where do goroutines wait on channels and locks? (needs runtime.SetBlockProfileRate)
mutex Who holds contended mutexes? (needs runtime.SetMutexProfileFraction)
threadcreate Where were OS threads created?

Newer Go versions also list a goroutineleak profile in pprof.Profiles(), aimed at goroutines that are blocked forever on channels or locks that nothing else can reach; check go doc runtime/pprof for its status in your version.

This program takes a CPU profile, a heap profile and a goroutine profile in memory. Profile output is a compressed binary format, so the program prints facts about them instead of the bytes:

Go Taking pprof profiles inside a program

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

In a real command-line tool you write the CPU profile to a file (os.Create("cpu.prof")) and flag-guard it. In a long-running server you don't write code at all: importing net/http/pprof for its side effect registers the profile endpoints on the default mux.

Go
import _ "net/http/pprof" // registers /debug/pprof/* on http.DefaultServeMux

func main() {
	go func() {
		// Listen on localhost only; these endpoints expose internals.
		log.Println(http.ListenAndServe("127.0.0.1:6060", nil))
	}()
	runServer()
}

Never expose /debug/pprof to the public internet. It reveals your code structure and lets anyone start expensive profiles. Bind it to localhost, or put it behind authentication on an internal port.

Reading a profile with go tool pprof

Fetch a 30-second CPU profile from the running server and open it:

Terminal
go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30

This drops you in an interactive prompt. The most useful commands are top, list and web. Sample output (the numbers are an example):

TEXT
(pprof) top 5
Showing nodes accounting for 2.40s, 80.00% of 3s total
      flat  flat%   sum%        cum   cum%
     1.10s 36.67% 36.67%      1.10s 36.67%  strings.(*Builder).WriteString
     0.60s 20.00% 56.67%      2.20s 73.33%  main.render
     0.40s 13.33% 70.00%      0.40s 13.33%  runtime.mallocgc
     0.20s  6.67% 76.67%      0.30s 10.00%  encoding/json.Marshal
     0.10s  3.33% 80.00%      0.10s  3.33%  main.handler

(pprof) list main.render

The two columns that matter:

  • flat is the time spent in the function's own code.
  • cum (cumulative) is the time in the function and everything it called.

A function with a large cum but a small flat is a coordinator; follow its callees. A function with a large flat is where the CPU is actually burned. list main.render prints the source of the function with the cost beside every line. For a visual overview, go tool pprof -http=:8081 cpu.prof opens a web UI with a call graph and a flame graph, where the width of a box is the time spent in that function and everything below it.

Heap and goroutine profiles use the same tool:

Terminal
go tool pprof -sample_index=alloc_space http://127.0.0.1:6060/debug/pprof/heap
go tool pprof http://127.0.0.1:6060/debug/pprof/goroutine

If the goroutine count keeps growing between two samples, you have a leak: goroutines that wait for something that never happens. The profile groups goroutines by stack, so the leaking stack is the one with a huge count. To compare two heap profiles taken minutes apart, use -base old.prof new.prof and look only at what grew.

When you profile a test or a benchmark (benchmarks and fuzzing), the flags do the file work for you:

Terminal
go test -bench=Render -cpuprofile=cpu.prof -memprofile=mem.prof
go tool pprof cpu.prof

Execution traces

A profile tells you how much time went where. An execution trace shows when: every goroutine's life, each time it was scheduled, blocked on a channel, waiting for the garbage collector or running a system call, laid out on a timeline per processor. It is the right tool for questions like "why is latency high when CPU use is low?" or "are my workers actually running in parallel?".

You record a trace with runtime/trace, and you can mark your own sections with tasks and regions so they show up by name in the viewer:

Go Recording an execution trace with tasks and regions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

wg.Go (Go 1.25) starts the goroutine and tracks it in the wait group. Each handle-request task becomes its own row in the viewer, and the load and render regions appear inside it. On your machine you save the trace to a file (or fetch /debug/pprof/trace?seconds=5 from a server) and open it with:

Terminal
go tool trace trace.out

The viewer opens in your browser with a timeline, a per-goroutine view, and tables for tasks, regions and blocking. Traces are large; record seconds, not minutes.

Stepping through code with Delve

When printing isn't enough, use a real debugger. Delve (dlv) understands goroutines, channels and Go's data structures far better than a general-purpose debugger. Most editors (VS Code with the Go extension, GoLand) use it behind their "Debug" button. From a terminal, a session looks like this (read-only; install with go install github.com/go-delve/delve/cmd/dlv@latest):

TEXT
$ dlv debug .
(dlv) break main.process
Breakpoint 1 set at 0x4a2f6a for main.process() ./main.go:13
(dlv) continue
> main.process() ./main.go:13 (hits goroutine(1):1 total:1) (PC: 0x4a2f6a)
(dlv) next
(dlv) print orders
[]string len: 2, cap: 2, ["pen","book"]
(dlv) print inv.stock
map[string]int nil
(dlv) goroutines
(dlv) stack
(dlv) quit

The commands you need most:

Command What it does
break file.go:42 / break pkg.Func Set a breakpoint (b for short)
continue (c) Run until the next breakpoint
next (n) / step (s) Run one line; step enters function calls
stepout Run until the current function returns
print expr (p) Evaluate and print an expression or variable
locals, args Print local variables or arguments
goroutines / goroutine 5 List goroutines / switch to one
stack (bt) Print the call stack of the current goroutine
condition 1 id == 7 Break only when an expression is true

Useful tricks: dlv test debugs a test, dlv attach <pid> attaches to a running process, and dlv debug -- args passes arguments to your program. If a variable shows as <optimized out>, the compiler removed it; build with -gcflags="all=-N -l" (which dlv debug does for you) to switch optimizations and inlining off.

GODEBUG: runtime switches

The GODEBUG environment variable turns on diagnostics in the runtime and the standard library without changing your code. Set it to a comma-separated list of name=value pairs:

Terminal
GODEBUG=gctrace=1 ./server              # one line per garbage collection
GODEBUG=schedtrace=1000 ./server        # scheduler summary every 1000 ms
GODEBUG=inittrace=1 ./server            # time spent in package init functions
GODEBUG=http2debug=1 ./server           # verbose HTTP/2 logging

Two related variables are worth knowing: GOTRACEBACK=all prints every goroutine on a crash (system and crash add more detail, and crash makes the process dump core), and GOMAXPROCS limits how many OS threads run Go code at the same time.

GODEBUG has a second job: it is the escape hatch for compatibility. When Go changes a behavior that might break existing programs, it ships a setting that restores the old behavior, and the version in your go.mod decides the default. For a single-file program you can also pin a setting in the source with a //go:debug line before the package clause. A concrete example is panic(nil): since Go 1.21 it panics with a *runtime.PanicNilError, so recover() never returns nil for a real panic. The setting panicnil=1 restores the old behavior. The first program asks for the modern behavior explicitly, the second one asks for the old one. (In a real module you would not write the first line at all: a go.mod that says go 1.21 or newer gives you the modern default. A single file compiled without a go.mod, like the editors on this page, falls back to the old defaults, which is why the first program states it.)

Go panic(nil) with the modern behavior

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Go The same program with //go:debug panicnil=1

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

With panicnil=0, recover returns a *runtime.PanicNilError, which has a real message and is easy to spot. With panicnil=1 it returns nil, which looks as if no panic happened at all. That is why the old behavior was changed. You will rarely need //go:debug, but knowing that these switches exist explains why upgrading Go is normally painless: old behavior stays available, and the toolchain tells you which settings your module uses (go list -f '{{.DefaultGODEBUG}}').

Choosing a tool

Symptom First thing to try
Wrong value Print the inputs and outputs with %+v / %q, or step through with Delve
Crash with a trace Read the trace top-down; find the first frame in your code
Hangs forever SIGQUIT or /debug/pprof/goroutine?debug=2; read the goroutine states
Slow CPU profile first; then heap/allocs if the GC shows up in the profile
Memory keeps growing Heap profile with -base; goroutine profile for leaked goroutines
High latency, low CPU Execution trace, block profile, mutex profile
Intermittent wrong result go test -race; see data races

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In a Go panic trace, which frame shows where the panic actually happened?

    A trace lists frames from the innermost call outward, so the first frame is where the panic was raised and each following frame is its caller.
  2. 2. In go tool pprof, what does the cum column of a function mean?

    flat is the function's own cost; cum (cumulative) includes the cost of its callees, so a coordinator function can have a large cum and a small flat.
  3. 3. A program prints fatal error: all goroutines are asleep - deadlock! Which statement is true?

    This is a fatal error, not a panic. It is reported only when all goroutines are blocked; a single leaked goroutine in a running program is not detected.
  4. 4. What does http.ListenAndServe(":6060", nil) with import _ "net/http/pprof" risk?

    The pprof handlers have no authentication. Listening on all interfaces would let anyone read internals and trigger expensive profiles.
  5. 5. Which tool shows WHEN goroutines ran or blocked on a timeline rather than just how much time each function used?

    Profiles aggregate samples; an execution trace records scheduling events over time, which is what you need for latency and parallelism questions.

Summary

  • Start cheap: print with %+v, %#v, %q and %T, add log.Lshortfile or slog, and remove the prints when you are done.
  • Read panic traces from the top; the first frame in your own code is usually where to look. A deadlock is a fatal error that names what the goroutine waits for.
  • runtime.Callers, runtime.Stack and debug.PrintStack let a running program inspect its own stacks and goroutines.
  • runtime/pprof and net/http/pprof produce CPU, heap, goroutine, block and mutex profiles; read them with go tool pprof (top, list, -http), and watch flat versus cum.
  • runtime/trace records a timeline with tasks and regions; open it with go tool trace.
  • Delve (dlv) gives you breakpoints, goroutine views and expression evaluation; GODEBUG and //go:debug switch runtime diagnostics and compatibility behavior.

Next: go vet, gofmt and Linting