Skip to content
Tutorials Center
My learning

Topics

Lesson 68 of 78 ยท Troubleshooting and Recipes

Repository Maintenance and Performance

Keep a Git repository small and fast: loose vs packed objects, git gc and git maintenance, finding large files in history, cleaning bloat, commit-graph and fetch.prune.

A new Git repository is tiny and instant. A ten-year-old one can be several gigabytes, take a minute to clone and make git status feel sluggish. Almost none of that is Git's fault: it is the result of what was put into the history, how the objects are stored, and which housekeeping jobs ran (or never ran) along the way.

This lesson is the maintenance manual. You will learn how Git stores objects on disk and why git gc exists, how to measure what is big, why deleting a file does not shrink a repository, how to really remove it, what git maintenance schedules for you, and which settings keep large repositories quick. Everything runs in the sandbox on tiny repositories, but the commands are exactly the ones you would use on a huge one.

Loose objects and packfiles

Every time you commit, Git writes new objects (blobs, trees, commits) into .git/objects as loose objects: one zlib-compressed file per object. That is fast to write and simple, but it is wasteful for storage, because a loose object never shares anything with similar objects. Edit one line in a 1,700-byte file six times and you store six nearly identical files.

A packfile fixes that. Git collects many objects into one file, sorts similar objects next to each other and stores most of them as deltas: "take that other object and apply these changes". An index file (.idx) lets Git find any object in the pack without reading the whole thing. Packing is what git gc does, and it is also what happens when you clone, fetch or push, because the network transfer is a packfile too. See Index, Packfiles and Garbage Collection for the file formats.

git count-objects -v shows how your objects are stored:

Bash Loose objects before and after git gc

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Before the gc, all 18 objects (six commits, six trees and six versions of the file) were loose: count: 18, occupying 72 KiB, because every loose file takes at least one filesystem block. After git gc there are count: 0 loose objects and one pack holding in-pack: 18. The last lines of verify-pack summarize how: 13 objects are stored whole, and 5 are deltas against another object. The five file versions that were deltas cost a few dozen bytes each instead of a full copy.

size for loose objects counts disk blocks, which is why a tiny repository "uses" 72 KiB. In -v mode size-pack is also in KiB. Add -H (git count-objects -vH) for human-readable units.

Git also packs for you without being asked. Commands such as git commit, git merge, git fetch and git pull end with git gc --auto, which checks two thresholds and only works when one is exceeded: more than gc.auto loose objects (default 6,700), or more than gc.autoPackLimit packs (default 50). Most of the time it does nothing and costs a few milliseconds.

What git gc does

git gc ("garbage collection") is a bundle of housekeeping steps:

  1. Pack refs: loose files under .git/refs are folded into one packed-refs file, which is faster to read when you have thousands of tags or branches.
  2. Repack: loose objects, and optionally existing packs, are rewritten into one well-compressed pack with good deltas.
  3. Prune: loose objects that are not reachable from any branch, tag, the reflog or the index, and are older than gc.pruneExpire (two weeks by default), are deleted.
  4. Expire the reflog: reflog entries older than gc.reflogExpire (90 days) and unreachable entries older than 30 days are dropped.
  5. Write the commit-graph (more on it below) when gc.writeCommitGraph is on, which is the default.

The two-week grace period for pruning is why you can usually recover "lost" commits (see The Reflog), and it is also why throwing history away does not shrink a repository immediately. Remember this; it matters in the next section.

You rarely run plain git gc by hand. The aggressive variant git gc --aggressive recomputes every delta from scratch with a deeper search. It can take hours on big repositories and the gain is small, so do not make it a habit: a normal git gc or a git repack -a -d once in a while is enough.

Measuring: what is big in this repository?

Before you clean anything, find out what is actually large. "The repository is big" almost always comes down to a handful of objects: a database dump, a compiled binary, a video, a node_modules folder that was committed once by mistake.

Three tools do most of the work:

  • git count-objects -vH for the totals.
  • git rev-list --objects --all, which lists every reachable object along with a path (for blobs and trees), piped into git cat-file --batch-check to get sizes.
  • git rev-list --disk-usage --objects ... to measure how many bytes a set of commits needs (Git 2.31 and later).

The following repository gets a large file committed and removed again. Watch what happens to the sizes:

Bash A deleted file still lives in history

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The working tree contains two small files, yet history still carries the 288,894-byte dump.sql blob (about 100 KB compressed) because the commit that added it is an ancestor of main. Git never forgets reachable history, and everyone who clones the repository downloads that blob. The git log -- path check tells you which commits are involved, and the sorted rev-list listing is how you find the culprit when you do not know it yet.

The %(rest) placeholder prints the path stored next to each object. The sort -rn sorts numerically from large to small. If you run this on a real project, bump head -3 to 20. For a file that was renamed or lives in many commits, the path shown is just one of the places the blob was found.

Cleaning history bloat

Now remove the dump from history for good. This is a history rewrite: every commit from the first one that contained the file gets a new hash, and so does every descendant. It is the right tool when a big file or a secret was committed, and the wrong tool for ordinary tidying. Read Amending and Filtering first, and coordinate with everyone who has clones, because they must re-clone or reset to the new history.

The sandbox build has the older git filter-branch, so we use it here. On your own machine, prefer the separate git filter-repo tool, which is faster and safer. The ideas are identical.

Bash Rewrite history and reclaim the space

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The numbers tell the story. Right after the rewrite the repository is not smaller at all: filter-branch kept a backup of the old branch under refs/original/, and the reflog still points at the old commits, so the blob stays reachable. Only after deleting the backup ref, expiring the reflog with --expire=now and running git gc --prune=now does the size fall from roughly 103 KB to under 1 KB.

git reflog expire --expire=now --all followed by git gc --prune=now destroys your safety net. Only run it after you have checked the rewritten history, and never in a repository that holds the only copy of unpushed work. On a hosting service you also have to remove the old commits from the remote: force-push every branch and tag, and ask the host to garbage collect, because pull requests and caches can keep old objects alive. If the file contained a secret, assume it is compromised and rotate it.

Prevention beats cleanup. Put build output and dependencies in .gitignore, keep large binary assets in Git LFS (the repository then stores small pointer files), and add a pre-commit or server-side check that rejects files over a size limit (the next sections and Hooks show how).

git maintenance: scheduled housekeeping

git gc is one big job. Since Git 2.30 there is a more modern tool: git maintenance, a set of small tasks that can run on their own schedule in the background. You can also run any of them by hand with --task:

Task What it does
gc the full git gc described above
commit-graph writes or updates the commit-graph file
prefetch fetches from every remote into hidden refs, so a later git fetch is fast
loose-objects packs loose objects in batches and deletes the ones already packed
incremental-repack combines small packs using a multi-pack-index, without a full rewrite
pack-refs packs loose refs
Bash Running maintenance tasks by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Two things are worth noticing. First, commit-graph wrote a commit-graphs directory under .git/objects/info, which we cover below. Second, loose-objects is a two-step job: the first run packed the 12 loose objects into a pack (in-pack: 12) but the loose copies were still there (count: 12); the second run's "prune packed" step removed them (count: 0). That is by design: the job packs in batches of 50,000 objects and deletes only objects that were already safely in a pack at the start of the run.

Letting Git run it in the background

On a real computer you do not run these by hand. git maintenance start registers the current repository and installs a scheduler (cron on Linux, launchd on macOS, Task Scheduler on Windows, or systemd timers) that runs hourly, daily and weekly tasks. The sandbox has no scheduler, so this part is read-only:

CONSOLE
$ git maintenance start
# registers the repository, sets maintenance.strategy=incremental if unset,
# and installs hourly/daily/weekly jobs in the system scheduler
$ git maintenance stop
# removes the schedule

The incremental strategy runs commit-graph and prefetch hourly, loose-objects and incremental-repack daily, and pack-refs weekly. It never runs a full gc, which is exactly why it is so gentle on large repositories. The list of repositories to maintain lives in your global config (maintenance.repo), and git maintenance register / unregister edit that list.

Fast fetches: fetch.prune and friends

A different kind of clutter builds up in refs/remotes. When someone deletes a branch on the server, your clone keeps the old remote-tracking branch (origin/feature) forever, and git branch -r fills up with ghosts. git fetch --prune removes remote-tracking branches that no longer exist on the remote, and fetch.prune=true makes that the default:

Bash Stale remote-tracking branches and fetch.prune

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A plain git fetch printed nothing and left origin/feature in place. git fetch --prune reported - [deleted] (none) -> origin/feature and the ghost disappeared. Setting fetch.prune (and optionally fetch.pruneTags) in your global config removes the chore. The same is true for git remote prune origin, which prunes without fetching. git maintenance register added the clone to the list of repositories the scheduler visits.

Other settings that speed up fetch and pull: fetch.parallel=0 (fetch from several remotes or submodules in parallel, using as many jobs as are useful), fetch.negotiationAlgorithm=skipping (finds common history faster in huge repositories), and fetch.writeCommitGraph=true (updates the commit-graph after every fetch).

Making big repositories feel fast

When a repository has hundreds of thousands of commits or files, the slow parts are not storage but walking: commands like git log, git status, git branch --contains and git merge-base must read many commits or stat many files. These features exist for exactly that:

  • commit-graph: a compact file with the parent links, generation numbers and timestamps of every commit, so history walks do not have to unpack each commit object. It makes git log --graph, git merge-base and git tag --contains many times faster on big histories. git maintenance keeps it current; enable core.commitGraph (the default) to read it.
  • multi-pack-index and reachability bitmaps: speed up object lookup and make clone and fetch cheap for servers. Hosting services manage these for you.
  • untracked cache and fsmonitor: git status must inspect every file in the working tree to find changes. core.untrackedCache=true caches directory listings, and core.fsmonitor=true (the built-in file system monitor, Windows and macOS) asks the operating system which files changed instead of scanning. On a repository with 200,000 files this can turn a ten-second git status into a fraction of a second.
  • Sparse checkout, shallow and partial clones: do not download or check out what you do not need: git clone --depth=1, --filter=blob:none, git sparse-checkout. See Large Repositories.
  • git config feature.manyFiles true: turns on a bundle of settings (index version 4, untracked cache) recommended for repositories with very many files.
  • Fewer, smaller refs: thousands of stale tags and branches slow down fetches and listings. Prune them, and let pack-refs run.
  • Avoid giant binary files in history entirely. Use Git LFS or an artifact store.

Also keep an eye on how often you rewrite packs: gc.auto and gc.autoPackLimit control the automatic thresholds. For a repository where git gc --auto has become annoying (big repository, slow disk), set gc.auto=0 and let the scheduled git maintenance do the work instead.

A maintenance checklist

If a repository feels slow or heavy, work through these steps in order:

  1. git count-objects -vH: many loose objects or many packs? Run git gc (or git maintenance run --task=gc).
  2. git rev-list --disk-usage --objects --all: is it much bigger than a fresh clone would be? Run the biggest-objects pipeline and find the culprits.
  3. Large files in history that do not belong there? Rewrite with git filter-repo, then expire reflogs and gc --prune=now, and make everyone re-clone.
  4. git status slow? Check for untracked build directories (ignore them), then try core.untrackedCache and core.fsmonitor.
  5. git log or git branch --contains slow? Make sure the commit-graph exists (git commit-graph verify).
  6. Cloning is slow for new teammates? Offer shallow, partial or sparse clones, and consider moving binaries to LFS.
  7. Turn on git maintenance start and fetch.prune, so none of this builds up again.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a packfile store that loose objects do not?

    A pack groups many objects into one file and stores most of them as deltas, which is why packed repositories are much smaller than the same objects stored loose. Git keeps every version, not just the newest.
  2. 2. You delete a 200 MB file in a new commit. Why is the repository still 200 MB larger than it needs to be?

    A commit only records a new snapshot. The old snapshot (and its blob) is still reachable from earlier commits, so clones keep downloading it until the history itself is rewritten.
  3. 3. After rewriting history with filter-branch the repository did not get smaller. What was still holding the old objects?

    filter-branch keeps a backup under refs/original, and the reflog points at the old commits. Remove the backup, expire the reflog, then run git gc --prune=now to drop the unreachable objects.
  4. 4. Which command measures how many bytes the commits reachable from all refs need?

    git rev-list --disk-usage prints the on-disk size of the listed objects. Add --objects to include trees and blobs and --all to cover every ref.
  5. 5. What does fetch.prune=true change?

    With fetch.prune, git fetch removes stale refs under refs/remotes for branches that no longer exist on the remote. Your own local branches are not touched.

Summary

  • Commits create loose objects; git gc (or the maintenance tasks) packs them into packfiles that store similar objects as deltas.
  • git count-objects -vH and git rev-list --disk-usage --objects --all measure a repository; rev-list --objects --all | cat-file --batch-check | sort finds the biggest objects.
  • Deleting a file does not shrink history. Removing a large blob needs a history rewrite (git filter-repo), then removing backup refs, git reflog expire --expire=now --all and git gc --prune=now.
  • git gc --auto runs on its own; git maintenance start schedules small, incremental tasks (commit-graph, prefetch, loose-objects, incremental-repack).
  • fetch.prune=true keeps git branch -r honest; commit-graph, untracked cache and fsmonitor keep big repositories fast.
  • Prevent bloat with .gitignore, Git LFS and size checks in hooks.

Next: Project: A Repository From Scratch