Git Reference ยท Concepts
Large files and big histories
Deal with big repositories in Git: why large files hurt, and how Git LFS, shallow clones, partial clones, sparse checkout and gc keep things fast.
Git stores every version of every file forever, and a clone normally downloads all of it. That works well for source code, but it breaks down with large binary files (videos, datasets, compiled artifacts) because binaries compress and delta poorly, and each change stores nearly a whole new copy. A repository that once held 50 MB can become several gigabytes after a few edits to one big file.
There are three separate problems and three families of tools:
- Big files in history: keep the content out of Git proper with Git LFS, or do not commit it at all.
- Long history: download less of it with a shallow clone (
--depth). - Many or large objects: download them lazily with a partial clone (
--filter) and check out only part of the tree with sparse checkout.
git clone --depth 1 URL # latest commit only git clone --filter=blob:none URL # all commits, blobs on demand git clone --filter=blob:limit=1m URL # skip blobs over 1 MB git sparse-checkout set docs src # only these folders on disk git lfs track "*.psd" # store matching files in LFS
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.
Which tool for which problem
| Problem | Tool | Trade-off |
|---|---|---|
| Huge binaries committed regularly | Git LFS | needs an LFS-aware server and git lfs installed |
| CI only needs the latest code | --depth 1 |
history commands (log, blame, describe) are limited |
| Developers need history, not every old blob | --filter=blob:none |
commands that need old blobs fetch them on demand, so need network |
| Monorepo, you work in one folder | sparse checkout | the rest stays in the object database but not on disk |
| Binary already committed by mistake | rewrite history (see below) | changes every later commit id |
Usage notes
- LFS replaces files with pointers. The repository holds a small text pointer (version, hash, size); the real content lives on the LFS server and is downloaded on checkout. Set tracking before adding the files;
git lfs migrate importcan convert existing history. - Shallow clones have no history beyond the cut-off, and
git fetch --deepen=Nor--unshallowrestores depth. They are great for CI builds, awkward for development. - Partial clones remember their "promisor" remote; commands can pause to download missing blobs. Combine
--filter=blob:nonewith sparse checkout for monorepos. - Removing a big file from history requires rewriting every commit that touched it (
git filter-repoorfilter-branch), and everyone must re-clone. Deleting it in a new commit shrinks nothing. See Rewriting history safely. - Housekeeping.
git gcandgit repack -a -dcompress loose objects;git count-objects -vHshows sizes. Large generated files belong in.gitignoreor in a package registry instead. - Server limits. Hosting services reject single files above a size limit (often 100 MB), another reason to use LFS or release assets.