Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 78 ยท Remotes and Collaboration

Submodules, Subtrees and Monorepos

Learn how to share code between Git repositories: submodule add, clone and update, common pitfalls, subtree merging, sparse checkout and the monorepo versus polyrepo trade-off.

Sooner or later a project needs code that lives somewhere else: a shared library, a vendored dependency, a design system used by five apps. Git offers several answers, and each one trades convenience against control. A submodule pins another repository at an exact commit. A subtree copies the other repository's files into yours and keeps merging them. A monorepo puts everything in one repository and then shrinks your working copy with sparse checkout. And often the best answer is none of these, but a normal package manager.

This lesson builds each option with local repositories so you can see exactly what Git records, and ends with a decision guide. It is one of the trickier topics in Git, so go slowly with the submodule section: nearly all the confusion comes from one fact. A submodule is a pointer to a commit in another repository, not a copy of its files.

Two sandbox details. First, there is no network, so the "other repository" is a bare repository on disk. Second, since Git 2.38.1, Git refuses to use the file transport for submodules unless you allow it (a security fix; local paths can be abused in hostile repositories). The examples therefore run git -c protocol.file.allow=always ... or set it with git config --global. With https or ssh URLs, as in real projects, you do not need this.

Submodules: a repository inside a repository

A submodule is a full, independent Git repository checked out inside a folder of the superproject (the outer repository). The superproject does not store the library's files. It stores:

  1. a line in .gitmodules (a tracked text file): the submodule's path and URL, and
  2. a special tree entry of mode 160000, called a gitlink, that holds the commit hash the submodule must be at.
TEXT
superproject commit
 |-- app.txt                      (normal file)
 |-- .gitmodules                  (path + URL of the submodule)
 `-- vendor/textlib   160000 -> 3a59e13...   (a commit of ANOTHER repository)
Bash Adding a submodule and looking at what Git stores

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

git submodule add cloned the library into vendor/textlib and staged two things: .gitmodules and the folder vendor/textlib. The recursive tree listing shows 160000 commit 3a59e13... vendor/textlib instead of a blob for the folder. The superproject's idea of the submodule (git rev-parse HEAD:vendor/textlib) is the same hash as the library's own HEAD, which is the whole mechanism. When someone later checks out an old superproject commit, Git knows which library version belongs to it.

The URL in .gitmodules can be relative (../textlib.git, as above), which keeps working when the whole group of repositories moves to a new host. The Git data of the submodule itself lives in the superproject's .git/modules/ folder.

Cloning and updating submodules

A plain git clone of a superproject gives you the submodule folder but not its content. The folder is empty until you initialise and fetch it. There are three ways:

  • git clone --recurse-submodules <url>: clone and fill everything in one go.
  • git submodule update --init (add --recursive for nested ones): after a normal clone.
  • git config --global submodule.recurse true: make pull, checkout and similar commands also update submodules.

When the library changes upstream, nothing happens in your project until you decide. git submodule update --remote moves the submodule to the latest commit of its tracked branch, which shows up in the superproject as a modification you must commit:

Bash Clone, initialise and update a submodule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In a plain clone the folder holds zero entries and git submodule status prints the pinned hash with a leading -, meaning "not initialised". After update --init the prefix is a space (clean). git submodule status has three prefixes worth memorising: - not initialised, + the checked-out commit differs from the one the superproject records, U merge conflict. After update --remote the status was +, git status -s showed M vendor/textlib, and git diff --submodule=log summarised the change as Submodule vendor/textlib 3a59e13..1051a0a: followed by > Library v2. Committing the modified folder pins the new version for everybody.

Notice the last command: inside the submodule, git status says HEAD detached at 1051a0a. A submodule is always checked out at a specific commit, not on a branch. If you want to change the library from inside the project, switch to a branch first (git switch main), commit there, push the library, then commit the new pointer in the superproject.

The classic pitfall: forgetting to push the submodule

The superproject only stores a hash. If you commit a new pointer but never push the library commit it refers to, the pointer is a promise nobody else can keep.

Bash The pointer nobody can resolve (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The clone of the superproject worked, then the submodule step failed with fatal: Fetched in submodule path 'vendor/textlib', but it did not contain <hash>. Direct fetching of that commit failed. and the line before it, not our ref, tells you the server has never seen that commit. The fix is to push the library commit from inside the submodule. To make mistakes like this impossible, tell Git to check:

Bash push --recurse-submodules=check refuses a dangling pointer (fails on purpose, then works)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

With push.recurseSubmodules set to check, the first push stopped before sending anything and named the submodule path whose commit "can not be found on any remote". After pushing the library first, the same command succeeded and a fresh recursive clone received lib v1 and lib v2. Setting it to on-demand makes Git push the submodule for you.

Other submodule habits and traps

  • Pull the superproject, then update submodules. git pull does not update submodules by default, so your files can silently disagree with the pointer. git submodule update --init --recursive after every pull (or submodule.recurse true) fixes it.
  • git status shows M vendor/textlib and you did nothing? Either the submodule's HEAD differs from the pointer (run git submodule update) or it has uncommitted changes (the status says -dirty in git diff).
  • Removing a submodule takes several steps (git submodule deinit -f <path>, git rm -f <path>, delete .git/modules/<path>); there is no single command.
  • CI must check out submodules explicitly (submodules: recursive in most CI configurations).
  • git submodule foreach 'git status -s' runs a command in every submodule.

Subtrees: copying another repository into yours

A subtree takes the opposite approach. The other project's files and history are merged into a subdirectory of your repository, so a normal clone of your repository has everything: no .gitmodules, no extra commands, no dangling pointers. Contributors do not need to know a subtree exists.

Git's contributed script git subtree (add, pull, push, split) automates this. It is not part of our sandbox build, so here is the read-only version of the commands you would run on a normal installation (labelled as such, because they do not run here):

Terminal
# read-only: needs the git-subtree script (included with most Git installs)
git subtree add  --prefix=vendor/textlib https://example.com/textlib.git main --squash
git subtree pull --prefix=vendor/textlib https://example.com/textlib.git main --squash
git subtree push --prefix=vendor/textlib https://example.com/textlib.git fixes   # send local changes back
git subtree split --prefix=vendor/textlib -b lib-only                          # extract history of the folder

Underneath, subtree uses plain Git features you can run: merge the other history with --allow-unrelated-histories, place its tree under a prefix with git read-tree --prefix, and later merge updates with the subtree merge option. Doing it by hand teaches you what the script does:

Bash A subtree by hand: read-tree and merge -X subtree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first merge -s ours --no-commit records the library history as an ancestor without changing any file (Git says Automatic merge went well; stopped before committing as requested). read-tree --prefix=vendor/textlib/ -u then drops the library's files into the prefix, and one commit seals it: the graph shows a merge with Library v1 as the second parent, and git ls-files lists vendor/textlib/lib.txt. When the library released v2, merge -X subtree=vendor/textlib matched the library's root to our prefix, so the change landed in vendor/textlib/lib.txt with full history and no special commands afterwards. Anyone who clones the app has all of it.

Submodule Subtree
Where the code lives in the other repository, pinned by hash copied into your repository
Clone needs extra steps yes (--recurse-submodules) no
Repository size small grows with the vendored history (use --squash)
Contributing changes back natural: it is a real repository possible, but harder (subtree push, split)
Typical pitfalls detached HEAD, unpushed pointers, forgetting update messy history, merge confusion
Best for a shared component several teams actively develop a dependency you vendor and rarely change

Alternatives: package managers and vendoring

Before you reach for either, ask whether you need Git at all. If the shared code is a library with versions, publish it as a package (npm, Composer, Maven, PyPI, a Go module) and depend on a version number. That gives you semantic versions, dependency resolution and caching, none of which submodules offer. Submodules and subtrees shine when the code is not packaged, needs to be edited alongside the main project, or must be pinned to an exact commit. Vendoring means committing a copy of the dependency's files to your repository, as one ordinary commit, with a note of where they came from. It is crude, but simple and robust, and it needs no special Git feature.

Monorepos and sparse checkout

A monorepo holds many projects (apps, services, libraries) in one repository. A polyrepo setup has one repository per project.

Monorepo Polyrepo
Cross-project change one atomic commit and PR several PRs, coordinated releases
Sharing code import directly, always in sync publish packages or use submodules
Tooling needs build-system support (affected-only builds, caching) each repo is simple
Access control coarse (everyone sees everything, unless the platform helps) per repository
Clone size and speed can grow huge small

Neither is "right". Monorepos pay off when many pieces change together and you have the tooling to build only what changed; polyrepos fit independent teams and release cycles.

The cost of a big monorepo is a big working directory. Sparse checkout lets you check out only the folders you work on, while the full history stays available:

Bash Sparse checkout in a monorepo

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

--cone mode takes whole directories (fast, and the default recommendation). After sparse-checkout set, the working tree held the root files plus services/billing and libs/common only; docs and the other services disappeared from disk but remain in Git, and disable brought them back. Combined with a partial clone (git clone --filter=blob:none, which also skips downloading file contents you do not need) this makes huge repositories pleasant to use; both are covered in Large Repositories: Shallow, Partial and Sparse.

Which should I use?

  • A normal package manager if the code can be a versioned package.
  • A submodule if it must stay a separate repository you also edit, and your team accepts the extra commands.
  • A subtree or plain vendoring if you want clones to just work and rarely change the code.
  • A monorepo (with sparse checkout and good tooling) if projects change together.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the superproject store for a submodule?

    The files stay in the other repository. The superproject records where it lives (.gitmodules) and which commit to use (the gitlink entry).
  2. 2. You cloned a project with submodules and the vendor folder is empty. What fixes it?

    A plain clone does not populate submodules. git submodule update --init (or cloning with --recurse-submodules) initialises and checks them out.
  3. 3. Inside a submodule, git status says 'HEAD detached at ...'. Why?

    Submodules are checked out at the recorded commit. To make changes, switch to a branch first, commit and push the library, then commit the updated pointer.
  4. 4. Which setting makes git push refuse to push a superproject whose submodule commits are not on any remote?

    With push.recurseSubmodules=check, Git verifies that the submodule commits referenced by the pushed commits exist on the submodule's remote and aborts otherwise.
  5. 5. What is the main advantage of a subtree over a submodule for people who clone your project?

    A subtree's files are part of your repository's own history, so git clone is enough. The price is a larger repository and trickier contribution back upstream.

Summary

  • A submodule is a pointer (gitlink, mode 160000) plus an entry in .gitmodules; the library lives in its own repository and is pinned to an exact commit.
  • Clone with --recurse-submodules, or run git submodule update --init --recursive; use git submodule update --remote to move to the latest library commit, then commit the new pointer.
  • Submodules are checked out detached. Push the library first, then the superproject, or set push.recurseSubmodules check.
  • A subtree merges another repository's files and history into a folder of yours; clones need no special steps, at the cost of a larger repository.
  • Prefer a package manager when the code can be versioned and published; vendoring is the simple fallback.
  • Monorepos keep everything in one repository; git sparse-checkout set --cone <dirs> and partial clones keep your working copy small.
  • On Git 2.38.1 and later, local-path submodules need protocol.file.allow=always; real network URLs do not.

Next: Worktrees and Multiple Working Directories