Skip to content
Tutorials Center
My learning

Topics

Lesson 53 of 78 ยท Customizing and Automating Git

Large Repositories: Shallow, Partial and Sparse

Make Git fast on huge repositories: shallow clones with --depth, partial clones with --filter, sparse-checkout, commit-graph, fsmonitor and Scalar. Local demos you can run.

Most repositories are small and Git feels instant. Then a project grows to ten years of history, hundreds of thousands of files and several gigabytes, and git clone takes half an hour and git status takes ten seconds. The fix is rarely "use something other than Git". Git has a set of features that let you download less, check out less and compute less. They fall into three groups:

  • Shallow clones download only recent history.
  • Partial clones download history but leave out big objects until they are needed.
  • Sparse checkout puts only some directories into your working folder.

Together with caches such as the commit-graph and a file-system monitor, they make even enormous monorepos pleasant. The sandbox has no network, but it can clone from a local path through a file:// URL, which behaves like a real server for these features, so everything below is runnable.

For --depth and --filter to take effect you must clone from a URL such as file:///path/to/repo or https://.... A plain path like ../repo uses a local shortcut that copies files directly and ignores both options (Git prints a warning).

Shallow clones: --depth

A normal clone downloads every commit ever made. git clone --depth N downloads only the latest N commits on each branch, plus everything those commits need. It is the standard trick for CI jobs, which only need to build the current commit.

Bash Shallow clone, deepen and unshallow

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The source had 5 commits, the shallow clone only 2, and git rev-parse --is-shallow-repository printed true. The file .git/shallow holds the hash of the boundary commit, whose parents were deliberately cut off (Git treats it as having none). git fetch --deepen 2 added two more commits, giving 4, and --unshallow fetched the rest: the repository is no longer shallow and has all 5 commits.

What you give up with a shallow clone:

  • git log, git blame and git bisect only see the fetched part of history.
  • Merging or rebasing with older history can fail with refusing to merge unrelated histories, because the common ancestor was never downloaded.
  • Tags and branches pointing at older commits are missing (--no-single-branch or --shallow-since=<date> give more control).
  • Pushing from a shallow clone works for ordinary commits, but some servers refuse unusual cases.

Use --depth 1 for CI builds, Docker image builds and one-off "just look at the code" downloads. For daily development prefer partial clones, because they keep full history.

Partial clones: --filter

Most of a large repository's size is blobs (file contents), often very old versions that nobody will ever open again. A partial clone downloads all commits and trees (so history queries work) but leaves out the blobs that match a filter. Git remembers where the repository came from, calls it a promisor remote, and fetches a missing object on demand the first time you need it, for example during checkout or git show.

Common filters:

Filter Leaves out
--filter=blob:none all file contents; they come on demand
--filter=blob:limit=1m files larger than 1 MB
--filter=tree:0 trees as well (smallest, but history operations fetch a lot afterwards)

The server must allow it. GitHub and GitLab do; for our local server we enable it with uploadpack.allowFilter.

Bash Partial clone with blob:none

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Right after the clone Git knew all 5 commits and their trees (25 objects in the pack), but all 15 blobs were missing, each marked with a ? by --missing=print. Checking out main made Git fetch only the 3 blobs of the latest versions that it needed to write the files: the missing count fell to 12 and the pack grew to 28 objects. The 12 old versions stay on the server until something asks for them. In a real repository that is millions of historical blobs you never download.

The trade-off: commands that touch old content (git blame, git log -p, git checkout of an old branch) trigger network requests, one batch at a time. Offline, they fail. Use git fetch --refetch --filter=... to change the filter later, and git sparse-checkout (next section) to keep working sets small.

Sparse checkout: only some folders

In a monorepo you may need only services/payments/ and the shared libs/. Sparse checkout keeps the whole repository in .git but writes only the chosen paths into your working tree. The modern way is cone mode: you list directories, Git includes everything under them plus the files directly at the root.

Bash Sparse checkout in cone mode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Before the command ls showed api docs web. After set web docs only docs and web exist on disk, and git sparse-checkout list prints those two names. Behind the scenes Git wrote three lines into .git/info/sparse-checkout: include the root files (/*), exclude all directories at the root (!/*/), then include /docs/ and /web/. Git still tracks all 15 files in the index; git ls-files -t marks the ones outside the cone with S, which is the skip-worktree flag. add widened the cone, and disable returned to a full checkout.

Sparse checkout saves disk space and checkout time and makes tools like editors and file watchers see fewer files. It does not by itself reduce what you download, which brings us to the powerful combination.

Partial clone plus sparse checkout

git clone --filter=blob:none --sparse <url> clones history without contents and checks out only the files in the root. Then you grow the working set directory by directory, and Git downloads exactly those blobs.

Bash Clone, then grow the working set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Right after the clone only README.md was on disk and 15 of the 16 blobs were still on the server. git sparse-checkout set web created the web folder and fetched just its 5 blobs, leaving 10 missing. All 6 commits were available the whole time, so git log worked offline. On a real monorepo this is the difference between a 20 GB and a 200 MB download.

Speeding up everyday commands

Even with the right files on disk, Git has to compare thousands of paths on every git status. Several caches help:

  • Commit-graph. A file (.git/objects/info/commit-graph) with precomputed commit relationships makes git log --graph, merge-base and reachability queries many times faster in big histories. Git writes it automatically on fetch and during maintenance.
  • File-system monitor. git config core.fsmonitor true starts a built-in daemon that watches the working tree so git status need not scan every file (Windows and macOS; Linux support depends on inotify). In the sandbox the daemon is not running.
  • Untracked cache. core.untrackedCache true remembers which directories have no untracked files.
  • Index v4 and feature.manyFiles. One switch that enables several settings for big working trees.
  • Background maintenance. git maintenance start schedules prefetching, commit-graph updates and packing with your system scheduler, so foreground commands stay quick (Repository Maintenance and Performance).
Bash Commit-graph and feature.manyFiles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

.git/objects/info was empty before and contains commit-graph afterwards. The graph is only a cache: queries return the same result with it disabled (5 both times), just slower on big histories. feature.manyFiles true turns on index version 4, the untracked cache and the file-system monitor where supported. On a five-commit repository none of this matters; it pays off with hundreds of thousands of files.

Scalar: all of this, configured for you

Microsoft's Scalar ships inside Git (the scalar command). It clones with --filter=blob:none and sparse checkout, enables the commit-graph, file-system monitor and background maintenance, and registers the repository for upkeep. It needs a real network and a scheduler, so these commands are read-only:

CONSOLE
$ scalar clone https://example.com/big/monorepo.git
$ cd monorepo/src
$ git sparse-checkout set services/payments libs
$ scalar list                  # repositories under Scalar's care
$ scalar register              # opt in an existing clone

Choosing the right tool

Situation Use
CI build of the latest commit git clone --depth 1
Daily work on a big repository with full history --filter=blob:none (partial clone)
Monorepo where you own one or two folders --filter=blob:none --sparse, then sparse-checkout set <dirs>
One huge binary file in history avoid committing it: Git LFS or a partial clone with blob:limit
Slow git status in a huge tree file-system monitor, untracked cache, feature.manyFiles
Repository carries years of junk cleaning history is a rewrite: see Fixing Common Git Mistakes and Amending, Splitting and Rewriting Commits

Remember that these are client-side choices. A repository that is huge because of how it was built still costs the server and every full clone, so keep binaries out of Git and split things that do not belong together (Submodules, Subtrees and Monorepos).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does git clone --depth 1 download?

    --depth N creates a shallow clone with only the latest N commits. Use git fetch --unshallow to get the full history later.
  2. 2. What is the difference between a partial clone and a shallow clone?

    Shallow limits how many commits you get. Partial limits which objects you get, and the promisor remote supplies missing ones when you need them.
  3. 3. After git sparse-checkout set web docs, what happens to the other directories?

    Sparse checkout only affects what is written to disk. All history and the index entries still exist, and git sparse-checkout add or disable brings the files back.
  4. 4. Why does git clone --depth 1 ../repo (a plain path) not give a shallow clone?

    For a local path Git copies objects directly and warns that --depth is ignored. A file:// URL goes through the normal transport.
  5. 5. Which combination downloads the least when you only work in one folder of a big monorepo?

    The filter avoids downloading file contents up front, and sparse checkout makes Git request only the blobs of the folders you check out.

Summary

  • git clone --depth N makes a shallow clone of recent history (--deepen, --unshallow extend it); it is ideal for CI but limits log, blame and merges.
  • --filter=blob:none (or blob:limit=<size>, tree:0) makes a partial clone: full history, contents fetched on demand from the promisor remote.
  • git sparse-checkout init --cone and set <dirs> write only some directories to disk; add and disable change or end it.
  • --filter=blob:none --sparse plus sparse-checkout set downloads only what you use. Use a file:// or real URL, not a bare path, for --depth and --filter.
  • The commit-graph, core.fsmonitor, the untracked cache and feature.manyFiles speed up daily commands; Scalar and git maintenance configure them automatically.
  • Fix the root cause when you can: keep large binaries out of Git, split unrelated projects.

Next: GitHub Basics