Skip to content
Tutorials Center
My learning

Topics

Lesson 98 of 147 ยท Modules and Packages

go generate, Workspaces and Vendoring

Go beyond the basics of go generate, go.work and vendor - directive variables, writing AST-based generators, freshness checks in CI, multi-module workspaces and the vendor directory.

Three tools in the Go toolchain deal with the shape of your build rather than your code: go generate for files that a program should write, go.work for developing several modules at once, and vendor/ for keeping your dependencies inside your repository. They are small, but they matter once a project has more than one author or more than one module. Project layout introduced them; modules and dependencies described the module system around them. This lesson goes deeper into how each one really works and where it goes wrong.

The sandbox cannot run the go command or touch your disk, so the commands appear as read-only blocks. The runnable programs re-create the logic of each tool, such as reading source with go/parser, expanding directives and resolving imports, so you can see the decisions the toolchain makes.

go generate in depth

go generate is a convenience for running commands that produce source files. It does nothing clever: it scans the Go files of the packages you name for lines that start with //go:generate and runs each command, in order, from the directory of the file. It is never run by go build, go test or go run, so the generated files must be committed to version control. That has a big benefit: people who download your module do not need the generator, or its dependencies, installed.

flowchart LR
    A["Source with<br/>//go:generate"] --> B["go generate ./..."]
    B --> C["Generator program<br/>(stringer, mockgen, your own)"]
    C --> D["Generated .go files<br/>(committed)"]
    D --> E["go build / go test"]

Generation is a separate, explicit step. The build only sees its results.

Variables, quoting and aliases

The command is split into arguments the way a shell would split them: spaces separate arguments, double quotes group words (and use Go string syntax). Before running, go generate replaces these variables, plus any environment variable written as $NAME or ${NAME}:

Variable Meaning
$GOFILE Base name of the file containing the directive.
$GOLINE Line number of the directive.
$GOPACKAGE Package name of that file.
$GOOS, $GOARCH Target platform of the toolchain.
$DOLLAR A literal $.

A special directive //go:generate -command NAME args... defines an alias that lasts until the end of the file. Everything in this scheme can be tried by writing a parser for it:

Go Expanding //go:generate directives

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Look at the output. The alias gen is replaced by go run ./internal/gen on later lines; $GOFILE and $GOPACKAGE become shape.go and shapes; the quoted file name stays one argument even though it contains a space; and ${DOLLAR}5 produces a literal $5. The last line has a space after the slashes, so it is an ordinary comment and is ignored. (The real tool also accepts Go raw strings and handles the details of error reporting, but the model is the same.)

Useful command line flags: go generate -n ./... prints what would run, -x prints each command as it runs, and -run regexp runs only directives whose command matches, for example go generate -run stringer ./.... If any command fails, go generate stops at that package with a non-zero status.

Pin your tools

A generator is a dependency like any other, and different versions can write different output. Do not rely on whatever happens to be installed. Two good ways:

Go
//go:generate go run golang.org/x/tools/cmd/[email protected] -type=Status

or, since Go 1.24, record the tool in go.mod once and call it through go tool:

Terminal
go get -tool golang.org/x/tools/cmd/stringer@latest   # adds a "tool" line to go.mod
Go
//go:generate go tool stringer -type=Status

Everyone who uses the repository then gets the same generator version, and go mod tidy and the dependency checks apply to it too.

Writing a generator with go/ast

The standard library has everything needed for a code generator: go/parser to read source, go/ast to walk it, text/template or a strings.Builder to write code, and go/format to run the result through gofmt. A good generator reads your real types, so the generated code can never disagree with them. This one finds a struct and writes methods that list its fields and turn a value into a map, without using reflection at run time:

Go A generator that reads a struct with go/ast

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The output is neatly formatted although the builder wrote everything unaligned: format.Source did that, and it also returns an error if the generated text is not valid Go, which catches template mistakes immediately. Add the Email field, and the freshly generated text differs from the committed one. That difference is the idea behind a CI check:

Terminal
go generate ./...
git diff --exit-code   # fails if running the generators changed anything

This keeps a pull request from forgetting to commit regenerated code. Follow two conventions. Begin generated files with a line matching // Code generated ... DO NOT EDIT. (a regular expression defined in the Go documentation); tools such as golint, gopls and code review sites recognize it, and ast.IsGenerated(file) tests for it in your own tools. And keep generated code in separate files so humans never edit them by hand.

When is generation the right tool? When the same boilerplate must follow a definition elsewhere (a schema, a protobuf file, a type), or when hand-writing is error prone, as for mocks and enum strings. When it is not: if generics, interfaces or a small helper can do the job, prefer them. Generated code is code your team must still read, review and build.

Workspaces with go.work

When one change spans two modules (say, a service and the library it uses) you need the service to build against your local copy of the library. The old way is a replace line in go.mod that you must remember to delete. A workspace file does the same without touching either module:

TEXT
work/
โ”œโ”€โ”€ go.work
โ”œโ”€โ”€ app/       module example.com/app, requires example.com/mathx v1.2.0
โ”‚   โ””โ”€โ”€ go.mod
โ””โ”€โ”€ mathx/     module example.com/mathx
    โ””โ”€โ”€ go.mod
Terminal
cd work
go work init ./app ./mathx
go work use ./another-module     # add one more (use -r . to scan the tree)
go work sync                     # copy workspace versions into each module's go.mod
go env GOWORK                    # which go.work is active (empty = none)
TEXT
go 1.27

use (
	./app
	./mathx
)

While go.work is active (the go command looks for it in the current directory and its parents), all use modules form one build. When app imports example.com/mathx/stats, the toolchain takes the package from ./mathx regardless of the version in app/go.mod. The file may also contain replace directives that apply to the whole workspace, and go work edit changes it from scripts. A go.work.sum file records extra checksums.

The module that provides an import is found by the longest matching module path among the workspace modules first, then among the required modules. This program imitates that rule:

Go Which module provides this import?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

With the workspace active, example.com/mathx/stats comes from ./mathx, a folder on your disk; with GOWORK=off it comes from version 1.2.0 in the module cache. Note that example.com/mathxtra does not match example.com/mathx: a module path matches only on whole path elements. Without a require, the unknown golang.org/x/text package cannot be resolved at all, which is the error you get from a missing dependency.

Some practical advice about workspaces:

  • A workspace is a development tool. go.work describes your machine's checkout, so most single-repository projects do not commit it. Monorepos with several modules sometimes do.
  • CI should build with GOWORK=off (or without a go.work file) to prove that each go.mod is complete on its own. Keep real require lines in each module even though the workspace overrides them locally.
  • Commands such as go test ./... match packages of the current module only. Name the modules explicitly in a workspace: go test example.com/app/... example.com/mathx/....
  • Publish the library first and update the require in the application afterwards. go work sync helps keep the versions aligned.

Vendoring

Vendoring copies the source of your dependencies into the repository, in a vendor/ directory:

Terminal
go mod vendor            # (re)create vendor/ from go.mod
go build ./...           # uses vendor/ automatically (go.mod says go 1.14 or newer)
go build -mod=mod ./...  # ignore vendor/ and use the module cache
go work vendor           # same idea for a workspace (Go 1.22+)

The directory contains the packages you actually import (not whole modules, and not their tests) plus a manifest, vendor/modules.txt, that records exactly which modules and versions are included. At build time the go command checks that this manifest agrees with go.mod; if you edit go.mod and forget to run go mod vendor, the build fails with an "inconsistent vendoring" error telling you to re-run it. This model of the check shows what is compared:

Go Is vendor/modules.txt consistent with go.mod?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Two things are wrong in this repository: x/text was upgraded in go.mod but not re-vendored, and cobra was added but never vendored. Re-running go mod vendor fixes both.

Should you vendor? Reasons to: builds must work offline or in an air-gapped network; reviewers want dependency changes visible in diffs; your company wants a copy under its own control. Reasons not to: the repository grows, merges produce noise, and the module proxy and go.sum already give reproducible, verified builds. If you vendor, check it in CI the same way as generated code: run go mod vendor and fail if git diff --exit-code vendor go.mod shows a change. Never edit files inside vendor/ by hand; the next go mod vendor would overwrite them. To patch a dependency, fork it and use a replace directive.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. When does //go:generate run?

    The directives are inert comments for the compiler. go generate runs them on demand, which is why generated files are checked in and users of your module need no generator.
  2. 2. What does $GOFILE expand to in a //go:generate command?

    $GOFILE is the base name of the source file with the directive. $GOPACKAGE and $GOLINE give its package and line number.
  3. 3. Which setup makes the CI pipeline fail when someone forgets to commit regenerated code?

    If the generators produce output different from what is committed, git diff reports changes and exits with a non-zero status, failing the build.
  4. 4. In a workspace, app imports example.com/mathx/stats and app/go.mod requires example.com/mathx v1.2.0. With go.work listing ./mathx, where does the package come from?

    Workspace modules take priority over required versions. That is the point of go.work, building against your local checkout. With GOWORK=off the v1.2.0 download would be used.
  5. 5. You edited go.mod to upgrade a dependency, and go build now reports inconsistent vendoring. What is the fix?

    The vendor manifest must match go.mod. go mod vendor regenerates both the copied sources and vendor/modules.txt. Never hand-edit vendored files.

Summary

  • go generate runs //go:generate commands on demand, from the file's directory, with $GOFILE, $GOPACKAGE, $GOLINE, $GOOS, $GOARCH and $DOLLAR expanded. -command defines aliases; -n, -x and -run help you inspect and filter.
  • Pin generator versions (go run pkg@version, or a tool directive and go tool), commit generated files with a Code generated ... DO NOT EDIT. header, and verify freshness in CI with go generate ./... && git diff --exit-code.
  • Generators can read real source with go/parser and go/ast, write code, and validate it with go/format.Source.
  • A go.work file with use entries builds several modules together against their local directories, without changing any go.mod. Build in CI with GOWORK=off.
  • go mod vendor copies the packages you import into vendor/ with a manifest, vendor/modules.txt, that must match go.mod. Use it for offline builds and auditing; never edit vendored files.

Next: Testing with the testing Package