Skip to content
Tutorials Center
My learning

Topics

Git Reference ยท Concepts

Large files and big histories

Deal with big repositories in Git: why large files hurt, and how Git LFS, shallow clones, partial clones, sparse checkout and gc keep things fast.

Git stores every version of every file forever, and a clone normally downloads all of it. That works well for source code, but it breaks down with large binary files (videos, datasets, compiled artifacts) because binaries compress and delta poorly, and each change stores nearly a whole new copy. A repository that once held 50 MB can become several gigabytes after a few edits to one big file.

There are three separate problems and three families of tools:

  • Big files in history: keep the content out of Git proper with Git LFS, or do not commit it at all.
  • Long history: download less of it with a shallow clone (--depth).
  • Many or large objects: download them lazily with a partial clone (--filter) and check out only part of the tree with sparse checkout.
Terminal
git clone --depth 1 URL                      # latest commit only
git clone --filter=blob:none URL             # all commits, blobs on demand
git clone --filter=blob:limit=1m URL         # skip blobs over 1 MB
git sparse-checkout set docs src             # only these folders on disk
git lfs track "*.psd"                        # store matching files in LFS

Try it

Bash Shallow clone and unshallow

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Bash Partial clone and sparse checkout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Which tool for which problem

Problem Tool Trade-off
Huge binaries committed regularly Git LFS needs an LFS-aware server and git lfs installed
CI only needs the latest code --depth 1 history commands (log, blame, describe) are limited
Developers need history, not every old blob --filter=blob:none commands that need old blobs fetch them on demand, so need network
Monorepo, you work in one folder sparse checkout the rest stays in the object database but not on disk
Binary already committed by mistake rewrite history (see below) changes every later commit id

Usage notes

  • LFS replaces files with pointers. The repository holds a small text pointer (version, hash, size); the real content lives on the LFS server and is downloaded on checkout. Set tracking before adding the files; git lfs migrate import can convert existing history.
  • Shallow clones have no history beyond the cut-off, and git fetch --deepen=N or --unshallow restores depth. They are great for CI builds, awkward for development.
  • Partial clones remember their "promisor" remote; commands can pause to download missing blobs. Combine --filter=blob:none with sparse checkout for monorepos.
  • Removing a big file from history requires rewriting every commit that touched it (git filter-repo or filter-branch), and everyone must re-clone. Deleting it in a new commit shrinks nothing. See Rewriting history safely.
  • Housekeeping. git gc and git repack -a -d compress loose objects; git count-objects -vH shows sizes. Large generated files belong in .gitignore or in a package registry instead.
  • Server limits. Hosting services reject single files above a size limit (often 100 MB), another reason to use LFS or release assets.