Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 78 ยท Git Internals

Scalar, fsmonitor and Commit-Graph for Big Repos

Speed up Git on huge repositories: the commit-graph file, Bloom filters, untracked cache, the fsmonitor daemon, background maintenance and the Scalar command.

Git is fast on small projects because it does very little work. On a repository with millions of commits and hundreds of thousands of files, "very little work" becomes a lot. git status has to check every tracked file to see whether it changed, git log --graph has to open and parse thousands of commit objects, and git fetch leaves behind more and more loose data. Microsoft ran into this when it moved Windows into Git, and the features they built for it are now in core Git.

This lesson explains the three performance ideas behind large-repository support: a precomputed commit-graph, a file-system monitor that tells Git which files changed, and background maintenance that keeps data structures tidy. Scalar is the command that switches them all on. We covered cloning less data in Large Repositories: Shallow, Partial and Sparse; here we make operations on the data you already have cheaper. The storage formats are explained in Git Objects and Index and Packfiles.

Why history queries are slow

A commit is a tiny object stored in a pack file. To answer "is commit A an ancestor of commit B?" or to draw git log --graph, Git walks parent links: read a commit, find its parents, read those, and so on. Each read means finding the object in a pack, decompressing it and parsing the text. Millions of such steps add up.

Worse, the walk needs an order. Git wants to visit commits from newest to oldest, and it uses the commit timestamp for that. Timestamps are only a hint (clocks differ, rebases keep old dates), so Git must walk beyond where it logically could stop to be safe.

The commit-graph file

The commit-graph is a compact binary file, .git/objects/info/commit-graph, that stores for every commit only what graph walks need: its tree ID, its parent positions, its commit date and a generation number. Because the file is dense and memory-mapped, Git reads thousands of commits without touching the pack files at all.

The generation number is the key trick. A commit with no parents has generation 1. Every other commit has a generation one higher than its highest-generation parent. So if commit A has a lower generation number than commit B, then A cannot be a descendant of B, and the walk can stop immediately instead of searching to the root.

flowchart LR
  A["A<br/>gen 1"] --> B["B<br/>gen 2"]
  B --> C["C<br/>gen 3"]
  B --> D["D<br/>gen 3"]
  C --> E["E<br/>gen 4"]
  D --> E

Each commit is one more than its highest parent, so E (4) can never be an ancestor of C (3).

Git can write the file by hand, and git commit-graph verify checks it. The next demonstration creates a repository with a few merges, writes the graph and shows its contents.

Bash Write and verify a commit-graph

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The graph never changes the answers, only how fast they come. Both log runs produce the same checksum. In Git 2.24 and later, fetch.writeCommitGraph and gc.writeCommitGraph keep the file up to date, and since it is only a cache, deleting it is always safe.

Bloom filters for path history

Commands such as git log -- src/parser.c or git blame ask "which commits touched this path?". Normally Git must compare the trees of each commit with its parent. With git commit-graph write --reachable --changed-paths, the graph also stores a small Bloom filter for each commit: a compact bit set summarising the paths it changed. A Bloom filter can say "definitely not changed" (skip this commit) or "maybe changed" (check properly), never a false negative. On large repositories, history for a single file gets many times faster. The setting commitGraph.changedPathsVersion=2 selects the improved hashing (the default in new versions).

Bash Changed-path Bloom filters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The untracked cache and fsmonitor

The commit-graph speeds up history. The next problem is the working tree: git status, git add and git diff need to know which files changed. By default Git compares the stat information (size, modification time) of every tracked file with what the index remembers, and then scans directories for untracked files. For 300,000 files that means 300,000 stat calls and thousands of directory reads, every time.

Two options reduce this:

  • core.untrackedCache remembers the modification time of each directory and skips re-reading the ones that did not change. It helps with untracked files, and you can turn it on with git config core.untrackedCache true (a test command, git update-index --test-untracked-cache, checks whether your file system supports it).
  • core.fsmonitor goes further and asks the operating system. Git ships a built-in file-system monitor daemon, git fsmonitor--daemon, which runs in the background, subscribes to file change notifications (inotify on Linux, FSEvents on macOS, ReadDirectoryChangesW on Windows) and records which paths changed since Git last asked. git status then checks only those paths.
sequenceDiagram
  participant FS as File system
  participant D as fsmonitor daemon
  participant G as git status
  FS->>D: notify: src/app.c changed
  G->>D: what changed since token T?
  D-->>G: only src/app.c
  G->>G: stat one file, not 300,000

The daemon turns "look at everything" into "look at what changed".

You enable the daemon per repository with one setting, and Git starts it on demand:

Terminal
# read-only: the daemon needs OS support and keeps running after the command
git config core.fsmonitor true
git config core.untrackedCache true
git fsmonitor--daemon status      # "fsmonitor-daemon is watching ..."
git fsmonitor--daemon stop        # stop it when you remove the repository

The daemon does not work everywhere: network drives are not supported, and in containers or sandboxes the notification API may be restricted. That is why it is shown read-only. Where it works, it can turn a status that takes many seconds into a fraction of a second. Without it, you can mark files as unlikely to change with git update-index --assume-unchanged or --skip-worktree, but those are manual and fragile, and are best avoided (see Troubleshooting for the traps).

Background maintenance

Large repositories also degrade silently: each fetch adds loose objects or small pack files, and the commit-graph becomes outdated. git maintenance runs the housekeeping tasks in a controlled way, in small, incremental steps that do not lock the repository for long, as opposed to one huge git gc.

Task What it does
commit-graph Updates the commit-graph file
prefetch Fetches new remote objects in the background into hidden refs, so the next fetch is instant
gc The classic full garbage collection (disabled in the recommended schedule)
loose-objects Packs loose objects in batches
incremental-repack Merges small packs into a multi-pack-index without rewriting everything

Run a task now with git maintenance run --task=commit-graph, or register the repository so the operating system's scheduler (cron, launchd or Task Scheduler) runs hourly, daily and weekly jobs: git maintenance start. Because registering edits your global Git configuration and installs a scheduled job, the example below only runs a task once, in a throwaway repository.

Bash Run one maintenance task by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Scalar: all of this with one command

Remembering a dozen settings is tedious, so Git includes Scalar, a small command (shipped with Git for Windows and macOS, and an optional build on Linux) that applies a recommended configuration for big repositories. scalar clone <url> does a partial clone with sparse checkout (see Large Repositories), and registers the repository for background maintenance. scalar register does the same for an existing clone.

Terminal
# read-only: Scalar is not available in our sandbox
scalar clone https://example.com/big/monorepo.git
cd monorepo/src
scalar list                  # repositories registered for maintenance
scalar reconfigure           # re-apply the recommended settings after a Git upgrade
scalar unregister            # stop background maintenance for this repo

The settings it applies include the following, which you can also set one by one:

INI
[core]
    fsmonitor = true
    untrackedCache = true
    commitGraph = true
[feature]
    manyFiles = true       # index version 4, untracked cache
[fetch]
    writeCommitGraph = true
[maintenance]
    strategy = incremental

For a mid-sized repository with a few thousand files you do not need any of this; the defaults are fine. When git status takes more than a second or two, or git log over history feels slow, try these steps in order: write the commit-graph, enable feature.manyFiles, enable fsmonitor, and finally consider scalar with partial clone and sparse checkout. Measure with time git status or GIT_TRACE2_PERF=1 before and after, so you know what helped. The routine care of a repository (gc, fsck, pruning) is described in Performance and Maintenance.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the generation number in the commit-graph tell Git?

    If A has a lower generation number than B, A cannot descend from B. Walks use this to stop early instead of searching to the root.
  2. 2. Which command writes the commit-graph including changed-path Bloom filters?

    --changed-paths stores a Bloom filter per commit that lets path-limited history commands skip commits that definitely did not touch the path.
  3. 3. What does the fsmonitor daemon do?

    The daemon subscribes to file change notifications. Git then checks only the changed paths instead of calling stat on every tracked file.
  4. 4. Why is git maintenance preferred over running git gc by hand on a huge repository?

    Maintenance splits housekeeping into tasks such as commit-graph, prefetch and incremental repack, which run on a schedule without one long, heavy gc.
  5. 5. What does scalar register do for an existing clone?

    Scalar applies settings like fsmonitor, untracked cache and the commit-graph, and registers the repo for scheduled maintenance.

Summary

  • The commit-graph (.git/objects/info/commit-graph) caches parent links, dates and generation numbers so history walks avoid parsing individual commit objects; git commit-graph write --reachable creates it and verify checks it.
  • --changed-paths adds Bloom filters, which make git log -- path much faster by skipping commits that definitely did not change the path.
  • core.untrackedCache and the built-in core.fsmonitor daemon cut the cost of git status by checking only files that changed.
  • git maintenance runs commit-graph, prefetch, loose-object and incremental-repack tasks in small steps; git maintenance start schedules them.
  • Scalar (scalar clone, scalar register) applies these settings plus partial clone and sparse checkout in one command.
  • Measure before and after; small repositories do not need any of this.

Next: Aliases and Everyday Productivity