Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 78 ยท Git Internals

The Index, Packfiles and Garbage Collection

Understand the Git index file, loose objects versus packfiles, delta compression, git gc, prune and repository size. Inspect everything with real commands.

Two questions come up once you know that Git stores objects as small files. First: where exactly does the staging area live, and what does it contain? Second: if every version of a file is stored, why are real repositories with years of history not enormous? The answers are the index and the packfile, and a housekeeping command called git gc that connects the two.

This lesson looks at both from the inside. You will read the index with git ls-files --stage, watch loose objects turn into a single compressed pack, see delta compression happen with git verify-pack, and learn what git prune and git maintenance do. The mental model from How Git Works and the object types from Git Objects are assumed.

The index: a proposal for the next commit

The index (also called the staging area or cache) is one binary file, .git/index. It holds a sorted list of every file that will be in the next commit: path, mode, blob ID, a stage number (0 normally, 1 to 3 during a conflict) and cached file-system data (size and timestamps). That cached data lets git status decide whether a file changed without reading it, just by comparing size and modification time to the saved values.

When you run git add file, Git writes the file's contents as a blob into the object database and records the blob's ID in the index. When you run git commit, Git turns the index into trees (write-tree) and creates a commit pointing at the root tree. The working tree is never read at commit time. This is why staging a file and then editing it again commits the staged version.

git ls-files --stage prints the index. Each line is mode blob-id stage path.

Bash Read the index with ls-files --stage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Look at the modes: 100644 for the plain file, 100755 for the executable (Git records only the executable bit, not full Unix permissions). The AM in the status output means "added to the index, and modified since" (see Status and Diff). The notation :notes.txt in git show means "the version in the index"; :0:notes.txt is the explicit form, and :2: / :3: pick the sides of a conflict.

Bash The index during a conflict has three stages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

During a conflict, Git keeps all three versions of the file in the index (stages 1, 2 and 3). Resolving means putting one final version at stage 0 with git add. That is why git add is also how you tell Git "this conflict is resolved". More in Resolving Merge Conflicts.

Loose objects

Every object you create starts as a loose object: one zlib-compressed file in .git/objects/xx/. Loose objects are quick to write, which is perfect while you work, but they have costs. A repository with 100,000 objects would have 100,000 files; each is compressed on its own, so near-identical file versions are stored as separate full copies; and sending them over a network one by one would be far too slow.

git count-objects -v counts them. In its output, count is the number of loose objects, size their disk usage in KiB, in-pack the objects stored inside packs and packs the number of packfiles.

Packfiles and delta compression

A packfile (.git/objects/pack/pack-<hash>.pack) stores many objects in one file, and an accompanying index file (.idx) lets Git find an object quickly by ID without scanning the pack. The key trick is delta compression: instead of storing a full copy of every version of a file, Git stores one version in full and the others as a "delta", a recipe saying "copy these byte ranges from the base object and insert these new bytes". Git does not care about file boundaries or history order when choosing bases; it sorts objects by type, name and size and tries similar ones as base candidates. Pack files are therefore usually far smaller than the sum of all file versions.

git gc (garbage collection) packs loose objects, and git repack does only the packing step. Git also runs gc by itself automatically when there are too many loose objects (more than about 6,700) or too many packs.

Bash From loose objects to one pack, with deltas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Before the gc there are 18 loose objects and no pack; afterwards the loose count is zero and all 18 live in one pack. The verify-pack -v listing shows that one version of big.txt (about 12,000 bytes, 1,227 after compression) is stored in full, and the five other versions are deltas against it that take only 18 bytes each. Git chooses the base by itself; it does not have to be the oldest version. chain length counts how many hops a delta needs to reach a full object; the default maximum depth is 50.

Compression in packs is why a clone of a repository is often much smaller than the sum of its history, and why big repositories with long histories are still manageable. Binary files that change entirely (videos, zip archives) delta badly and make a repository grow by their full size every time you commit a new version.

Under the hood: pack-objects, pack size and repairing a corrupt pack

Every pack, whether made by git gc, git repack, git push or git clone, is produced by one plumbing command: git pack-objects. It reads a list of object IDs on standard input, chooses delta bases and writes a .pack plus its .idx (and, in recent versions, a .rev reverse index). You can run it yourself: git rev-list --objects --all | git pack-objects name packs every reachable object, and git pack-objects --stdout writes the pack to standard output, which is exactly what the server does during a fetch. Related commands are git index-pack (builds the .idx for a pack you received), git unpack-objects (explodes a pack into loose objects) and git verify-pack (checks a pack).

What decides the size of a pack

  • How similar the objects are. Deltas only help between similar versions of a file, such as consecutive commits of one source file. Compressed formats (zip, jpg, mp4) change completely with every edit and gain almost nothing.
  • --window=N (default 10): how many neighbouring objects Git tries as a delta base. A bigger window finds better bases but costs time and memory; --window=0 switches deltas off entirely.
  • --depth=N (default 50): the longest chain of deltas allowed. Deeper chains save space but make reading an old object slower, because Git must replay each step.
  • Packing everything together. One big pack can delta across all history, while many small packs (for example one per push) cannot. This is why git gc --aggressive and git repack -a -d -f --window=250 --depth=50 can shrink a repository, at a high CPU cost; see Performance and Maintenance.
  • Garbage. Unreachable objects are kept for the grace period of gc.pruneExpire (two weeks), so a pack can hold more than you need until the next prune.
Bash pack-objects, delta settings and a damaged pack

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first half shows that the .pack, .idx and .rev files are all pack-objects output, and that with deltas off all 12 objects are stored whole (non delta: 12 objects), while the default run stored three of them as deltas. The second half is the emergency drill.

Repairing a corrupt pack

A corrupt pack usually comes from a failing disk, an interrupted copy or a full disk. Git detects it because every pack ends with a SHA-1/SHA-256 checksum and every object has its own CRC in the index. The symptoms are error: inflate: data stream error, pack checksum mismatch, bad object or missing blob, and git fsck exits with a non-zero code. Work through these steps and copy the whole repository folder first, because every repair step changes it:

  1. Find out how bad it is. git verify-pack -v .git/objects/pack/*.idx and git fsck --full list the damaged objects.
  2. If another copy exists, use it. Any clone, backup or remote has the objects. Move the damaged pack out of .git/objects/pack/ (keep the .idx with it), delete stale caches such as .git/objects/info/commit-graph, and fetch again. git fetch --refetch <remote> downloads everything even though Git thinks it already has it. This is what the example does.
  3. If you have no other copy, salvage what is readable: git unpack-objects -r < damaged.pack (run from a repository without these objects) writes every object it can still decode as a loose object. Then run git fsck; if only a few blobs are lost, restore those files from your working directory with git hash-object -w file, which recreates the blob with the right ID, or reset the branch to an older commit that is still complete, using the reflog.
  4. Verify and clean up. git fsck --full must exit with 0. Then git gc to repack.

Prevention is cheaper than repair: keep a second copy (a remote or git bundle create backup.bundle --all), and turn on transfer.fsckObjects so that fetches and pushes check incoming objects. For more recovery recipes see Troubleshooting.

What git gc does

git gc is a bundle of housekeeping steps:

  1. Pack refs into packed-refs (References).
  2. Expire reflog entries older than the limits (90 days reachable, 30 days unreachable by default), see the Reflog.
  3. Repack: combine loose objects and existing packs into a new, tightly delta-compressed pack. Objects that nothing reaches are kept loose for the grace period instead of going into the pack.
  4. Prune loose objects that are unreachable and older than gc.pruneExpire (two weeks by default). The grace period protects commits that another running Git process may have just created.
  5. Update auxiliary data such as the commit-graph and prune old worktree info.

You rarely need to call it yourself, because hosting services and Git's gc --auto do. Run it manually when a repository feels slow or after you removed big files from history.

The next example shows the two-step nature of deleting data. A commit that is no longer reachable survives a normal gc, because of the grace period, but disappears when you shorten the period with --prune=now.

Bash Unreachable commits and git prune

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Two things protected the commit: the reflog (removed with reflog expire) and the grace period (bypassed by --prune=now). After that the object is physically deleted and cannot be recovered. Use these options only on purpose, for example after rewriting history to remove a leaked secret, and know that every clone that already has the object keeps it.

Understanding repository size

Clones become large for a handful of reasons, in this order of likelihood: someone committed a big binary or a generated file; many versions of a medium-size binary were committed; history contains an old data dump. Git remembers everything, so deleting a file in a new commit does not shrink the repository; the old blob stays reachable through history.

Useful commands:

  • git count-objects -vH shows sizes in human-readable units.
  • git rev-list --objects --all lists every reachable object with its path, which you can combine with git cat-file --batch-check to find the biggest blobs.
  • git gc --aggressive spends much more time searching for good deltas. It is rarely worth it.
  • git maintenance (from Git 2.29) runs gc, commit-graph, prefetch and loose-object tasks on a schedule without blocking you. git maintenance start registers a background schedule. The task list is gc, commit-graph, prefetch, loose-objects, incremental-repack and pack-refs.
Bash Find the biggest objects in a repository

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

This is the standard recipe for finding bloat: list all reachable objects, ask for their type and size, sort by size. The 30,000-byte file shows up although the latest commit has no data.bin. Removing it from history requires a rewrite (see Amending, Splitting and Rewriting Commits). For routine care see Repository Maintenance and Performance.

The commit-graph and other caches

Walking history by opening commit objects one at a time gets slow in large repositories. git commit-graph write --reachable stores commit parents, dates and generation numbers in .git/objects/info/commit-graph, so commands such as log --graph, merge-base and branch --contains can answer without parsing commits. git gc and git maintenance keep it up to date automatically in modern versions. Other caches are the multi-pack-index and bitmaps used by servers to speed up fetches; you normally never touch them.

Bash Write and check a commit-graph

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The graph is purely an accelerator: delete the file and everything still works, only slower.

Practical rules

  • Do not run git gc --prune=now or reflog expire casually; they remove your safety net.
  • Do not commit generated files, build outputs or large binaries. Use .gitignore (ignore rules) and, for assets that must be versioned, see Git LFS.
  • A pack is immutable. Each gc produces a new one and deletes the old, so disk usage can briefly double during a repack.
  • A repository where count-objects reports thousands of loose objects can be sped up with git gc.
  • The index is local and not versioned. If it ever gets corrupted, git reset rebuilds it from HEAD without touching your files.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the index (.git/index) contain?

    The index is the staging area: path, mode, blob ID and stage for every file in the proposed next commit, plus cached stat data.
  2. 2. During a merge conflict, what do index stages 1, 2 and 3 represent?

    Stage 1 holds the merge base, stage 2 your side (HEAD) and stage 3 the side being merged. Adding a resolved file replaces them with stage 0.
  3. 3. Why is a packfile smaller than the same objects stored loose?

    Delta compression stores one full copy and small differences for similar versions. No data is lost; every object can be rebuilt exactly.
  4. 4. You reset a branch and want unreachable commits permanently removed right now. Which is closest to what is needed?

    The reflog and the two-week grace period keep unreachable objects. Expiring the reflog and pruning with now deletes them for good.
  5. 5. You deleted a 200 MB file in a new commit but the repository is still huge. Why?

    History keeps every version of every file. Shrinking the repository requires rewriting history so the blob is no longer reachable, then pruning.

Summary

  • The index is a binary file listing path, mode, blob ID and stage for the next commit; git ls-files --stage prints it, and :path / :N:path read blobs from it.
  • Objects start loose (one compressed file each) and are later combined into packfiles with an .idx lookup file.
  • Packs use delta compression: similar versions are stored as small differences from a base object.
  • git gc packs refs and objects, expires reflogs and prunes unreachable objects after a grace period (two weeks by default); --prune=now removes them immediately.
  • git count-objects -v, verify-pack -v and rev-list --objects --all | cat-file --batch-check reveal repository size and bloat.
  • Deleting a file does not shrink a repository; only rewriting history and pruning does.
  • git maintenance and the commit-graph keep large repositories fast.

Next: Transfer Protocols and How Fetch and Push Work