Skip to content
Tutorials Center
My learning

Topics

Lesson 69 of 78 ยท Troubleshooting and Recipes

Removing Secrets and Large Files from History

Found a password or a huge file in your Git history? Learn to find it, rotate the credential, rewrite history with filter-repo or filter-branch and handle the force-push.

Sooner or later someone commits a file that should never have been committed: an .env with a database password, a cloud access key, a private key, or a 400 MB video that makes every clone painfully slow. The reflex is to delete the file in a new commit. That does not fix anything. The file is still in every earlier commit, and anyone with a clone (or a CI cache, or a fork) still has it.

This lesson gives you the order of operations for a leaked secret: contain the damage first, then clean the history, then deal with the consequences for your team. You will find the offending file, rewrite history to remove it, confirm that it is really gone, and see exactly what a force-push does to other clones. History rewriting builds on Amending and Filtering History, Reflog and Push and Pull.

Rule one: assume the secret is compromised

Rewriting history removes a secret from your repository. It cannot remove it from copies that already exist: clones, forks, pull request refs on the hosting platform, CI logs, build caches, or crawlers that scrape public repositories within minutes of a push. So the order is:

flowchart TD
  A["Secret committed and pushed"] --> B["1. Revoke and rotate the credential"]
  B --> C["2. Check logs for misuse"]
  C --> D["3. Rewrite history to remove it"]
  D --> E["4. Force-push, expire old objects"]
  E --> F["5. Tell the team to re-clone or reset"]
  F --> G["6. Prevent a repeat: .gitignore, hooks, scanners"]

Rotating the credential is the real fix; rewriting history is cleanup.

Rotate first. Create a new password or key, switch your systems to it, and revoke the old one. Then look at the provider's audit logs for use you do not recognize. Only after this is the leaked value harmless. If the secret was never pushed (it exists only in local commits), rewriting is enough and nobody else can have it.

Finding what to remove

Know exactly which paths and which commits you are dealing with. git log -S (the "pickaxe") finds commits where the number of occurrences of a string changed, and --all searches every branch and tag. See Blame and Search for more of these tools.

For large files, list the biggest blobs in the whole history. The pipeline lists every object reachable from any ref, asks Git for its size, and sorts:

Bash Find a leaked secret and the biggest files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The pickaxe finds two commits: the one that added the password and the one that removed the file. The git show proves that deleting the file in a later commit hid nothing. And the size listing shows dump.sql as the heavy object that would bloat every clone.

Rewriting history

To remove a file from every commit, you need a tool that rewrites all commits that contain it. Each rewritten commit gets a new hash, and so do all of its descendants, which is why this is a team event, not a private tweak.

git filter-repo is the tool Git itself recommends. It is a separate Python script (install it with your package manager or pip install git-filter-repo), it is fast, and it is safe by default: it works on a fresh clone and removes the origin remote so that you cannot push by accident. Its common calls are short:

Terminal
# read-only: git filter-repo is not part of core Git or of our sandbox
git clone --no-local https://example.com/team/app.git app-clean && cd app-clean

# remove one file or folder from all of history
git filter-repo --invert-paths --path .env

# remove every blob bigger than 10 MB
git filter-repo --strip-blobs-bigger-than 10M

# replace secret strings inside files (one 'old==>new' rule per line)
echo 'hunter2==>***REMOVED***' > replacements.txt
git filter-repo --replace-text replacements.txt

BFG Repo-Cleaner (a Java program) does the same for the two most common cases (--delete-files .env, --strip-blobs-bigger-than 10M) and is popular for its speed on huge repositories. Both tools protect your latest commit by default and expect you to have removed the file there first.

The old built-in git filter-branch can do the same job with no extra install. It is slow, has pitfalls, and its own manual recommends against it, but it is there on every machine, and it shows the mechanism clearly: it replays each commit, applying a filter to the tree. This example removes .env and dump.sql from all commits on all branches and tags:

Bash Rewrite history to remove a file, then prove it is gone

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Three details matter in that example. --index-filter edits the index of each commit without checking files out, so it is much faster than --tree-filter. --prune-empty drops commits that became empty (here none, because app.py changed in each). --tag-name-filter cat rewrites the tags, so v0.1 points to a clean commit. And in the end filter-branch leaves a backup under refs/original/, the reflog still remembers the old tips, and gc does not delete recent objects, so you have to remove the backup refs, expire the reflog and prune before the secret is physically gone from the .git folder.

Run a rewrite on a fresh clone, never on your only copy, and check the result before you push it. Also check every branch, tag, stash and refs/pull/* ref: anything not matched by --all is still carrying the secret.

Force-push and its consequences

A rewritten history shares no commits with the remote's version after the first changed commit, so a normal git push is rejected. You have to overwrite the remote branches. Prefer --force-with-lease, which refuses to overwrite if someone pushed in the meantime (see Push and Pull):

Terminal
# read-only: after the rewrite, in the cleaned clone
git remote add origin https://example.com/team/app.git
git push --force-with-lease --all
git push --force --tags

On the hosting platform you may need to turn off branch protection briefly, ask the support to purge cached pull request refs and views of the old commits, and re-run CI. Forks and clones made earlier keep the secret; only rotating the credential protects you from them.

For your colleagues, the old and new histories have different commits. If they simply pull, Git tries to merge the two and brings the old commits (and the secret) back. Here is what that looks like from a teammate's side:

Bash What a teammate sees after your force-push

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

After git fetch, Bob's status says the branches have diverged. The right response is to save any local work (for example with git stash or a backup branch), then git reset --hard origin/main, or to clone again. If Bob has unpushed commits based on the old history, he can move them onto the new one with git rebase --onto origin/main <old-base> <his-branch> (see Rebasing). Announce the rewrite in advance: tell everyone to push their work first, and give the exact commands.

Preventing the next leak

  • Add secrets and local config to .gitignore from day one, and commit an .env.example with fake values.
  • Use a secret manager or CI secrets, and read them from the environment.
  • Install a pre-commit hook or scanner (gitleaks, trufflehog, detect-secrets) so keys are caught before commit (Git Hooks), and enable the secret scanning of your hosting platform (push protection blocks known key formats).
  • Keep large binaries out of Git, or track them with Git LFS. Note that adding LFS later does not shrink history; you must rewrite it too (git lfs migrate import).
  • When the problem is only size, consider whether a history rewrite is worth the disruption. A partial or shallow clone can hide the weight without touching commits.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You pushed an API key by mistake. What is the first thing to do?

    Once pushed, assume the key is copied. Rotating it makes the leaked value useless; cleaning the history is secondary.
  2. 2. Why does 'git rm .env' followed by a commit not remove the password from the repository?

    Each commit is a snapshot. Removing the file only affects later snapshots; the older snapshots still contain it until history is rewritten.
  3. 3. After a successful filter-branch, why might the secret still be inside the .git folder?

    filter-branch keeps a backup of the old tips, the reflog still points at the old commits, and gc keeps recent objects. Delete the backup refs, expire the reflog and run gc --prune=now.
  4. 4. A teammate runs a plain git pull after your force-push. What is the risk?

    Both histories contain different commits, so Git merges them, and the old commits reappear. He should reset to the rewritten branch or re-clone.
  5. 5. Why is --force-with-lease preferred over --force?

    --force-with-lease checks that the remote branch is still where you last saw it, so you do not destroy someone else's new work.

Summary

  • Deleting a file in a new commit does not remove it from history. A pushed secret must be treated as compromised: revoke and rotate it first, then check logs.
  • Find the culprit with git log --all -S'text' and list big blobs with git rev-list --objects --all | git cat-file --batch-check.
  • Rewrite history with git filter-repo (preferred: --invert-paths --path, --strip-blobs-bigger-than, --replace-text), BFG, or the slow built-in git filter-branch --index-filter. Work on a fresh clone and rewrite tags and every branch.
  • Delete refs/original, expire the reflog and run git gc --prune=now to purge the objects locally.
  • Publish with git push --force-with-lease; ask the platform to purge cached refs. Forks and old clones still hold the secret.
  • Teammates must reset or re-clone; a plain pull can resurrect the old history. Prevent repeats with .gitignore, secret scanners and hooks.

Next: Project: Your First Repository