Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 78 ยท Git Internals

Transfer Protocols and How Fetch and Push Work

See how Git talks to remotes: URLs and protocols, refspecs, what fetch and push transfer, shallow and partial clones, and offline transfer with git bundle.

When you type git fetch or git push, two Git programs have a short conversation: one asks "what refs do you have?", the other answers, they work out which objects the receiver is missing, and a single compressed pack of exactly those objects travels across. Everything else, such as origin/main appearing and moving, is bookkeeping on your side that follows a rule called a refspec.

This lesson explains that conversation and the rules around it. You will learn the URL and protocol options, read and write refspecs, look at what a clone really configures, make shallow and partial clones, and move history around without any server by using bundles. In our sandbox there is no network, so every demo uses local repositories (a bare repository stands in for the server). The commands and refspecs behave identically over HTTPS or SSH. Background on remotes and tracking branches is in Remotes and push, pull and Tracking Branches.

Remote URLs and protocols

A remote is just a name for a URL in .git/config. Git picks the transport from the URL:

URL form Transport Typical use
https://host/team/repo.git HTTP(S), "smart" protocol Public and private hosting, passwords or tokens
git@host:team/repo.git or ssh://git@host/team/repo.git SSH Developers with keys
/path/to/repo.git or ../repo.git Local, Git runs the other side as a child process Same machine, backups, tests
file:///path/to/repo.git Same as local, but through the full protocol Testing what a real transfer would do
git://host/repo.git The Git daemon, unauthenticated, read-only in practice Rare today

On all of them the work is done by two helper programs: on the side that has the data, git upload-pack (serves fetch and clone), and for pushes git receive-pack (accepts new objects and updates refs). SSH simply runs those programs on the server through the shell; HTTPS wraps the same conversation in web requests (git-upload-pack and git-receive-pack endpoints), with a web server or hosting platform in front. SSH and HTTPS need credentials and a network, so the examples below are read-only for them; the file:// form lets us watch the same protocol locally.

A plain local path like ../repo.git takes a shortcut: Git may hard-link or copy object files directly, and ignores options like --depth. Use file:// URLs when you want to test behaviour that real transport has.

The conversation: what happens during a fetch

The modern wire format is protocol version 2. In simplified form, a fetch goes like this:

TEXT
 client                                         server (upload-pack)
   |                                                  |
   |  <-- capabilities: ls-refs, fetch, object-format |
   |  --> command=ls-refs (ask for refs/heads, tags)  |
   |  <-- list of "<id> <refname>"                    |
   |                                                  |
   |  --> command=fetch                               |
   |      want <ids of refs I do not have>            |
   |      have <ids of commits I already have>        |
   |      done                                        |
   |  <-- packfile with only the missing objects      |

The client sends wants (tips it needs) and haves (commits it already owns, walked back from its own tips). The server finds the newest commits both sides share and builds a thin pack of everything reachable from the wants but not from the common commits. Because it uses the delta compression you saw in packfiles, a fetch after a small change transfers a small pack. The client then stores the pack (or unpacks it into loose objects when it is tiny) and finally updates its remote-tracking refs.

You can watch the first part of this talk with GIT_TRACE_PACKET=1. Here we ask a local server for its refs.

Bash Watch the ref advertisement with GIT_TRACE_PACKET

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The server first announces its version and capabilities (ls-refs, fetch, object-format=sha1), then answers the client's ls-refs command with one <id> <refname> line per ref. ls-remote shows the same list without fetching anything, which is the quickest way to check "does the remote have my branch, and at what commit?" without touching your repository.

What a clone sets up

git clone <url> dir is a convenience for four steps: create an empty repository, add the remote origin, fetch everything, check out the default branch. The result is visible in .git/config.

Bash What git clone writes into the config

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The line remote.origin.fetch = +refs/heads/*:refs/remotes/origin/* is the single most important line here. It is the default refspec: it says how remote refs are mapped onto your local names. It is why a branch called feature on the server shows up as origin/feature locally, and the leading + allows the local copy to be moved even when it is not a fast-forward (remote-tracking refs just mirror the server).

Refspecs

A refspec has the form [+]<source>:<destination>. For fetch, the source is a ref on the remote and the destination is a ref on your machine. For push it is the other way round: source is yours, destination is on the remote. A * wildcard may appear once on each side.

Refspec Meaning
+refs/heads/*:refs/remotes/origin/* Default fetch: every branch, mirrored as a remote-tracking branch
refs/tags/*:refs/tags/* Fetch all tags into the same names
main:copy-of-main Fetch the remote main into a local branch called copy-of-main
HEAD:refs/heads/renamed Push: create or update renamed on the remote from your current commit
:old-branch Push with an empty source = delete old-branch on the remote
refs/pull/*/head:refs/remotes/origin/pr/* Hosting-specific refs, such as GitHub pull requests

git push origin feature is shorthand for git push origin refs/heads/feature:refs/heads/feature, and git push origin --delete old is shorthand for git push origin :old. You can also put your own fetch = lines in the config, for example to mirror only one branch: fetch = +refs/heads/main:refs/remotes/origin/main.

Bash Fetch and push with explicit refspecs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Fetching into refs/remotes/box/* created a second set of remote-tracking names. (origin/* and the tag v1 appeared too: Git also updates the remote's normal tracking refs when it fetches from it, and it follows tags that point into the fetched history.) Fetching main:copy-of-main made an ordinary local branch from the remote. Note how FETCH_HEAD records what the last fetch retrieved; git pull is literally git fetch followed by merging (or rebasing onto) that. The last two pushes created and deleted a branch on the server using only a refspec. Git refuses a non-fast-forward update to a destination unless the refspec starts with + or you use --force; --force-with-lease is the safer version that checks the remote is still where you last saw it.

What push does

A push is the mirror image. The client connects to git receive-pack on the server, which advertises its current refs. The client works out which commits the server lacks, builds a pack and sends it together with a list of commands: "move refs/heads/main from <old> to <new>". The server checks each command (fast-forward only unless forced, hooks such as pre-receive, protected branch rules on hosting platforms), stores the objects, updates the refs and reports the result for every ref separately. That per-ref report is why one push can partly succeed.

The old value in each command is the compare-and-swap you met in References: the server only moves a ref that still has the value the client believed it had. This is also what makes --force-with-lease possible. After a successful push the client updates its own refs/remotes/origin/<branch>.

Shallow clones

A shallow clone cuts history at a chosen depth. git clone --depth 1 <url> fetches the newest commit with its tree and nothing before it. It is perfect for CI jobs and for downloading a project to build it, because the transfer is a fraction of the full history. The cut-off commits are listed in .git/shallow; to Git they are commits without parents.

Bash A shallow clone and unshallow

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Pitfalls: shallow clones cannot show the full git log, git blame stops at the boundary, and merges against old commits can fail with "no merge base". --shallow-since=<date> and --shallow-exclude=<ref> cut by date or tag instead of count. For everyday work, prefer a partial clone.

Partial clones

A partial clone downloads all commits and trees, but leaves out the large file contents (blobs) until they are needed. git clone --filter=blob:none <url> skips all blobs; Git fetches the ones it needs, for example for the files of the first checkout, one batch at a time. You keep complete history for log, branch and merge-base, and pay for old file versions only when you look at them. --filter=blob:limit=1m skips only blobs over one megabyte. The server must allow filters (uploadpack.allowFilter, enabled on the big hosting sites). The clone becomes a promisor repository: its remote promises to supply missing objects on demand.

Bash A blobless partial clone

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In the example, the clone downloaded 4 commits and 4 trees, but only the latest f.txt (needed for the checkout). The other three file versions are "missing". Showing HEAD~2:f.txt triggered a transparent extra fetch, after which one fewer object is missing. Partial clones are covered in more depth in Large Repositories.

Your own Git server over SSH

You do not need GitHub or GitLab to share code. Any Linux machine with SSH and Git installed is a perfectly good Git server, and for a small team or a home lab it takes ten minutes. The idea is that a Git "server" is just bare repositories in a directory plus a way to log in.

  1. Create a user. sudo adduser --disabled-password git makes an account named git. Everyone connects as this one user, and is told apart by their SSH key.
  2. Authorize keys. Each developer sends their public key (~/.ssh/id_ed25519.pub). Append it to /home/git/.ssh/authorized_keys, one key per line. The directory must be mode 700 and the file 600, owned by git.
  3. Create bare repositories. sudo -u git git init --bare /srv/git/project.git. Bare means no working tree: only the contents of .git, which is what a shared repository needs (see Remotes).
  4. Use it. On any client: git clone git@server:/srv/git/project.git, or for an existing project git remote add origin git@server:/srv/git/project.git followed by git push -u origin main.

Do not hand out shell access

With this setup everyone who has a key can also open a normal shell as git. Restrict the account to Git commands by setting its login shell to git-shell (it ships with Git; add it to /etc/shells first, then sudo chsh git -s $(which git-shell)). git-shell allows only git-receive-pack, git-upload-pack and git-upload-archive, which are exactly the programs that git push, git fetch, git clone and git archive start on the server. Anything else is refused. You can also restrict individual keys in authorized_keys with options in front of the key:

TEXT
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAAC3Nza... ada@laptop

If you create a ~/git-shell-commands directory with executable scripts in it, git-shell offers an interactive menu of only those commands, which is how people add a list or new-repo helper without opening a real shell.

Other ways to serve the same repositories

  • Git daemon (git daemon --base-path=/srv/git --export-all) serves git:// URLs with no authentication on port 9418. It is read-only unless you enable daemon.receivepack, which is dangerous, so use it only for public, read-only mirrors on a trusted network.
  • HTTP(S) works through a web server running git http-backend (a CGI program) and gives you passwords or tokens and TLS, which is friendlier behind firewalls. This is the "smart HTTP" protocol from the table above.
  • git instaweb starts a temporary web interface (using gitweb, a CGI script, and a server such as lighttpd) so you can browse a repository in a browser. Treat it as a quick local viewer; real teams install Gitea, GitLab or similar, which add accounts, permissions, issues and pull requests on top of the same SSH and HTTP protocols (see GitLab, Bitbucket and Self-Hosting).

The sandbox cannot run SSH, but the part that matters for security, git-shell and the bare repository, does work here. The server side of an SSH clone is nothing more than running git-upload-pack in the repository directory, which we can do by hand:

Bash A bare repository and git-shell, without the network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The grep line counts the lines of the server's reference advertisement that mention refs/heads/main (two: the first line names it as the HEAD target, the second lists the branch). The server announces its branches, then the client says what it wants, exactly as in the fetch conversation earlier in this lesson. The interactive login is refused because there is no git-shell-commands directory. Next steps for a real server are SSH hardening (keys only, no root login; see Signing, Credentials and Security), backups with git bundle create --all, and git config core.sharedRepository group if several Unix users must write to the same repository.

Bundles: moving history without a server

A bundle is a single file that contains a pack plus a list of refs. It lets you hand a repository (or just the new commits) to someone through email, a USB stick or a file share, which is useful when two machines cannot reach each other or the network is air-gapped. git bundle create, verify, list-heads and unbundle handle the file, and git clone, git fetch and git pull accept a bundle path in place of a URL.

Bash Create, verify and clone from a bundle

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

An incremental bundle created with a range (shipped..main) is small, but it requires the receiver to already have the commit shipped points at; git bundle verify tells you right away if it does not. Because a bundle is a normal pack, everything about refspecs and tracking in this lesson applies to it as well. Bundles also make a handy single-file backup of a repository: git bundle create backup.bundle --all.

Practical advice

  • To see what a remote has without fetching: git ls-remote origin. To see what a fetch would change: git fetch --dry-run.
  • git fetch is always safe: it only adds objects and moves remote-tracking refs. git pull also merges into your branch, so prefer fetch followed by a look at git log HEAD..origin/main when you are unsure.
  • Use SSH or HTTPS with a token for real remotes. Credential helpers and security are the topic of Signing, Credentials and Security.
  • For slow transfers, try --depth, --filter=blob:none, or git config fetch.prune true to drop deleted remote branches (Branch Management).
  • If a push is rejected, Git tells you why (non-fast-forward, hook declined, protected branch). Read the message; the fix list is in Troubleshooting.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the refspec +refs/heads/:refs/remotes/origin/ tell a fetch to do?

    The part before the colon names the remote refs, the part after names the local refs they update, and + allows non-fast-forward updates to those mirror refs.
  2. 2. What is the effect of git push origin :old-branch?

    An empty source means pushing 'nothing' to that destination, which removes the ref. git push origin --delete old-branch is the readable form.
  3. 3. During a fetch, what are 'haves' and 'wants'?

    The negotiation lets the server compute a minimal pack: everything reachable from the wants but not from the common commits.
  4. 4. You want a CI job to download only the latest commit of a project as fast as possible. Which option fits?

    --depth 1 makes a shallow clone with just the newest commit. Remember that with a plain local path Git ignores it; use a real URL or file://.
  5. 5. When is a git bundle useful?

    A bundle packs objects and refs into one file that clone and fetch can read like a remote, perfect for offline transfer and backups.

Summary

  • A remote is a named URL; transports are HTTPS, SSH, local paths, file:// and the rarely used git://. On the other side upload-pack serves fetches and receive-pack accepts pushes.
  • Fetch negotiates wants and haves, then receives one thin pack with only the missing objects; push sends a pack plus ref update commands, which the server checks per ref.
  • Refspecs [+]src:dst map refs between repositories: fetch pulls remote src into local dst, push sends local src to remote dst; an empty source deletes.
  • git clone writes remote.origin.url, the default fetch refspec and the upstream of the checked-out branch.
  • Shallow clones (--depth) cut history; partial clones (--filter=blob:none) keep history but fetch file contents on demand.
  • git bundle packs history into a file for offline transfer, incremental updates and backups.
  • git ls-remote and GIT_TRACE_PACKET=1 let you inspect what a remote offers and what is said on the wire.

Next: How Merge, Rebase and Diff Work Inside