Skip to content
Tutorials Center
My learning

Topics

Lesson 124 of 147 ยท Advanced Go

The Go Scheduler and Runtime Internals

Understand how Go runs goroutines - the G, M and P model, work stealing, preemption, the netpoller, blocking syscalls, stack growth and the runtime and runtime/debug packages.

You can start a hundred thousand goroutines in Go and the program stays responsive, even on a laptop with four cores. Operating-system threads cannot do that: each one costs megabytes of memory and a trip through the kernel to switch. The reason Go can is its own scheduler, a piece of the runtime that multiplexes many cheap goroutines onto a few real threads.

You do not need to understand the scheduler to write concurrent Go (goroutines and channels hide it well). But knowing the model explains behavior that otherwise looks magical: why a blocked goroutine costs almost nothing, why GOMAXPROCS exists, why a tight loop used to freeze programs, and why the profiler reports what it does. This lesson is a guided tour, with small programs that show the effects without relying on timing or exact numbers.

The G, M and P model

The scheduler works with three kinds of objects. Their one-letter names appear in runtime source code, in traces and in GODEBUG output:

Letter Stands for What it is
G Goroutine The code to run plus its own small stack and saved registers
M Machine An operating-system thread, which actually executes instructions
P Processor A scheduling context: a license to run Go code, with a local queue of runnable goroutines

An M must hold a P to run Go code. The number of Ps is GOMAXPROCS, which defaults to the number of CPU cores. So with 8 cores at most 8 goroutines run Go code at the same moment, no matter how many exist. The others are either runnable (waiting in a queue for a P) or blocked (waiting for a channel, a lock, a timer or I/O, and not in any queue).

TEXT
      P0 (queue: G7 G8)   P1 (queue: G9)   P2 (queue: empty)
       |                   |                |
      M0 runs G1          M1 runs G2       M2 runs G3

      global queue: G10 G11        blocked: G4 (channel), G5 (timer), G6 (network)

Each P has its own run queue, so most scheduling decisions need no lock shared with other cores. That is a large part of why the scheduler scales.

How a goroutine gets picked

When an M with a P finishes or blocks its current goroutine, it looks for the next one in a fixed order, roughly:

  1. Every 61st scheduling round, check the global queue, so that goroutines sitting there are never starved.
  2. Take the next goroutine from the P's local queue.
  3. If that is empty, check the global queue.
  4. Check goroutines that have become ready because of the network poller (explained below).
  5. Steal work: pick another P at random and take half of its queue.
  6. If nothing is found anywhere, the M parks (sleeps) until work appears.

That fifth step is work stealing. It means you do not have to balance load by hand: if you spawn 1,000 goroutines from one place, idle Ps pull them away from the busy one.

Seeing goroutines without counting on timing

runtime.NumGoroutine() returns how many goroutines currently exist. The program below starts a thousand goroutines that all block on one channel. Blocked goroutines use a few kilobytes each and cost the scheduler nothing while they wait:

Go A thousand blocked goroutines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The three lines are at start: 1 (just main), while blocked: 1001 and at the end: 1. The count is a useful leak detector: in a long-running server a NumGoroutine value that only ever grows usually means goroutines are stuck forever, a bug covered in concurrency patterns and testing concurrent code.

GOMAXPROCS

GOMAXPROCS limits how many Ps exist, which limits how many goroutines run Go code in parallel. You can set it with the environment variable of the same name or with runtime.GOMAXPROCS(n), which returns the previous value, and GOMAXPROCS(0) just reports the current one. Two points are worth knowing:

  • Since Go 1.25 the default also respects a Linux container CPU limit (cgroup quota), and the runtime adjusts it if the limit changes while the program runs. Before that, a service limited to two CPUs in Kubernetes would still start with the machine's full core count and waste effort on throttling. Older versions used the automaxprocs library as a workaround.
  • Setting it to 1 makes a program concurrent but not parallel. Goroutines still interleave, but only one executes at a time. That is handy for experiments, and is what this page's runner mostly gives you anyway.

Because the number depends on the machine, never print it as if it were a result, and never write code that works only for a particular value.

Go Cooperating on one P with runtime.Gosched

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

runtime.Gosched() politely hands the P back to the scheduler, which puts the calling goroutine at the back of the global queue and runs something else. Here it lets the worker run even though only one P exists. The output is procs now 1: true, worker: running and main: worker finished, yielded at least once: true. In normal code you almost never need Gosched; channels and locks yield automatically. Busy-waiting on an atomic flag like this is a demonstration, not a recommendation: use a channel or sync.WaitGroup instead.

Preemption: why a tight loop no longer freezes the program

A scheduler needs a way to take the CPU from a goroutine that does not give it up. Early Go versions were only cooperative: a goroutine could be switched out at function calls, channel operations and similar points. A loop such as for {} with no calls never reached one, and with GOMAXPROCS=1 it froze everything else.

Go 1.14 added asynchronous preemption. A background thread named sysmon notices a goroutine that has run for more than about 10 ms and sends its thread a signal, whose handler can stop the goroutine at almost any instruction (at a safe point where the garbage collector can still understand the stack). The next program has a goroutine that spins forever with no function calls, yet main still gets to run:

Go A busy loop cannot starve other goroutines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Both lines print, and the program ends. On a Go version before 1.14 with one P, main could be stuck forever. The timer fires, main becomes runnable, and the spinner is preempted to let it in. Preemption is about fairness, not speed: it makes latency predictable. It does not make a busy loop a good idea.

Blocking, system calls and the netpoller

What happens when a goroutine blocks depends on why:

  • Channel, mutex, timer, select: the runtime parks the goroutine (removes it from the queue, remembers what it waits for) and the M immediately runs another G. No thread is blocked. The goroutine is made runnable again when the event happens.
  • Network I/O: sockets are set to non-blocking mode and registered with the operating system's readiness API (epoll on Linux, kqueue on macOS and BSD, IOCP on Windows). A goroutine that calls conn.Read with no data is parked, and the netpoller wakes it when the socket becomes readable. This is the reason a Go server can hold hundreds of thousands of idle connections with just a handful of threads, while still letting you write simple blocking-style code.
  • Blocking system calls (reading a regular file, some DNS lookups, cgo calls): these really do block a thread. The runtime lets the M keep its P for a very short time. If the call is still running, sysmon hands the P off to another M (starting a new thread if needed), so other goroutines continue. When the call returns, the original M tries to reacquire a P, or else puts its goroutine in the global queue and parks itself.

You can watch the first and second behaviors together. The program below parks 200 goroutines inside real loopback TCP reads. If every blocked read held a thread, the program would create hundreds of threads. It does not:

Go Many blocked network reads, few threads

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The output is three lines with true, true and all readers released. The runtime created a small, machine-dependent number of threads (the exact number is deliberately not printed) for 200 goroutines waiting on sockets. The same program with a thread per connection would be much heavier. Networking in more depth: TCP and protocols.

A goroutine blocked in a system call that cannot be made non-blocking, such as reading a regular file, occupies a real thread. A program that does thousands of simultaneous slow file or cgo operations can therefore create thousands of threads. The runtime caps the total at 10,000 by default (debug.SetMaxThreads) and crashes beyond that. If you need that many parallel blocking operations, bound them with a worker pool or semaphore.

Goroutine stacks grow on demand

Each goroutine starts with a small stack (historically 2 KB; since Go 1.19 the starting size adapts to the average observed by the program). Function prologues check whether the stack has room. If not, the runtime allocates a bigger one (double the size), copies the old contents over, rewrites pointers that point into the stack, and continues. Stacks also shrink during garbage collection when they are mostly empty. That is why you can recurse deeply without configuring anything:

Go A deep recursion on a tiny starting stack

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The goroutine recurses 200,000 calls deep, so the output is recursion depth reached: 200000 and default max stack in bytes: 1000000000 (the 1 GB limit). Compare this with C, where the call stack is a fixed size, typically 8 MB, chosen by the operating system when the thread is created. Infinite recursion in Go ends with fatal error: stack overflow, which cannot be recovered. Stack copying is also why Go forbids holding pointers into a stack from outside it, and why escape analysis matters.

The runtime and runtime/debug packages

You will use a few functions from these packages in daily work:

Function Purpose
runtime.NumGoroutine() Current number of goroutines (leak detector)
runtime.Gosched() Yield the processor
runtime.GC() Force a garbage collection (garbage collection)
runtime.ReadMemStats(&m) Heap size, allocation counts, GC statistics
runtime.Caller(skip) / runtime.Callers File and line of the caller (used by loggers)
runtime.LockOSThread() Pin the goroutine to its current thread (needed by some GUI and OpenGL libraries)
runtime.Goexit() End the current goroutine after running deferred calls
debug.Stack() / debug.PrintStack() The current goroutine's stack trace as bytes / on stderr
debug.SetGCPercent, debug.SetMemoryLimit Tune the collector
debug.SetMaxThreads, debug.SetMaxStack Safety limits
debug.ReadBuildInfo() Module and version information embedded in the binary

Two of them make nice demonstrations. runtime.Caller finds where a line of code lives, and debug.Stack returns a trace you can log inside a recovered panic:

Go runtime.Caller, Goexit and stack traces

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

where() reports main.go:34, the line of the call. The recovered panic carries the runtime's message assignment to entry in nil map, and the stack text includes the function name riskyWork. Goexit stops only its own goroutine, runs its deferred calls, and leaves the rest of the program alone.

Observing the scheduler yourself

Two built-in tools show the scheduler at work on your own machine:

Terminal
# Print scheduler statistics every 1000 ms to standard error
GODEBUG=schedtrace=1000 ./myprogram

# Record an execution trace and open it in the browser
go test -trace trace.out   # or runtime/trace in a program
go tool trace trace.out

schedtrace prints lines such as gomaxprocs=8 idleprocs=6 threads=12 runqueue=0 [3 0 0 0 1 0 0 0]: the number of Ps, how many are idle, the thread count, the global queue length and each P's local queue. The execution trace (see profiling and debugging) is a timeline where you can see each goroutine run, block and wake, which is the clearest way to learn how your program behaves.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the G, M, P model, what does GOMAXPROCS limit?

    GOMAXPROCS is the number of Ps. An M needs a P to run Go code, so at most that many goroutines run in parallel. There can be far more goroutines (blocked or queued) and more threads (for example in blocking system calls).
  2. 2. A goroutine calls conn.Read on a TCP connection with no data waiting. What happens?

    Network sockets are non-blocking and registered with the OS readiness mechanism, so waiting costs a parked goroutine, not a thread.
  3. 3. What problem did asynchronous preemption (Go 1.14) solve?

    Before Go 1.14 switching required cooperation at function calls and similar points. Signals now let the runtime interrupt such loops.
  4. 4. What does work stealing do?

    When a P's local queue is empty, its M picks another P and steals half of its runnable goroutines, balancing load automatically.
  5. 5. What is true about the starting stack of a goroutine?

    Goroutine stacks start small and are grown (and shrunk) automatically, which is why a program can run huge numbers of goroutines and recurse deeply.

Summary

  • The scheduler runs many goroutines (G) on a few threads (M). A P is the permission and run queue needed to execute Go code; there are GOMAXPROCS of them.
  • Idle Ps steal work from busy ones, and the global queue is checked regularly, so load balances itself without your help.
  • Channel, lock and timer waits park the goroutine, not the thread. Network I/O is handled by the netpoller. Real blocking system calls get their own thread while the P is handed to another M.
  • Asynchronous preemption (Go 1.14) lets the runtime interrupt tight loops, so one goroutine cannot hog a P forever.
  • Stacks start small and grow by copying; the 1 GB limit stops runaway recursion with an unrecoverable stack overflow.
  • Use runtime.NumGoroutine, runtime.Caller, debug.Stack, GODEBUG=schedtrace and the execution trace to observe the runtime, and never build logic on exact timings, thread counts or scheduling order.

Next: Performance Tuning shows how to measure first and then remove the costs that actually matter.