Skip to content
Tutorials Center
My learning

Topics

Lesson 62 of 78 ยท GitHub and Platforms

Git with Jenkins and Other CI Servers

How CI servers like Jenkins, TeamCity and GitLab CI use Git: webhooks versus polling, what a pipeline checkout really does, shallow clones, clean workspaces and credentials.

A continuous integration (CI) server is a program that watches a repository, fetches new commits and runs your build and tests. Jenkins is the best-known self-hosted example; TeamCity, Bamboo, Buildkite, Azure Pipelines and GitLab CI follow the same pattern. The previous lesson, GitHub Actions: CI for Your Repository, shows a hosted system where the checkout is hidden inside an action. With a self-managed server you see the Git layer clearly, and a lot of "CI is flaky" stories are really Git stories: a stale workspace, a shallow clone without the tag you need, a detached HEAD that surprises a script.

In this lesson you will learn the two ways a CI server finds out about a change (webhooks and polling), what a pipeline checkout does step by step, how to keep workspaces clean and fast, and how to handle credentials safely. Jenkins itself cannot run here, so its configuration files are read-only, but the Git behavior underneath all of it runs in the sandbox.

The loop every CI server runs

flowchart LR
  A["Developer pushes"] --> B["Git server"]
  B -->|"webhook (push)"| C["CI server"]
  C -.->|"polling (pull)"| B
  C --> D["Checkout the commit"]
  D --> E["Build and test"]
  E --> F["Report status to the commit"]

A push reaches the CI server by webhook or by polling; the result is reported back against the commit.

Every CI system has to answer three questions: when should I run, which commit do I build, and where do I put the result. Git answers all three: a new commit hash means there is work, the hash identifies exactly what to build, and the hash is also where status checks attach on the hosting service.

Webhooks: the server tells you

A webhook is an HTTP request that the Git server sends to the CI server when something happens. It is fast (builds start within seconds) and cheap (no idle traffic). In Jenkins you enable the trigger "GitHub hook trigger for GITScm polling" or the generic "Build when a change is pushed" option, and then register the CI server's URL in the repository settings of GitHub, GitLab or Bitbucket. The payload tells Jenkins which repository and branch changed, and Jenkins then fetches the commits itself. The hook just means "look now".

On a plain Git server, you can build the same thing yourself with a post-receive hook, which runs after a push has been accepted and gets one line per updated ref on standard input. This runnable example sends the "webhook" to a log file instead of an HTTP address, and only for pushes to main:

Bash A post-receive hook that notifies CI

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The push to main appended a notification; the push to experiment printed remote: post-receive: refs/heads/experiment ignored and notified no one. A real hook would curl the CI server's URL, and hosted services send a richer JSON payload with the repository, the branch and the commit list. Keep webhook receivers protected by a secret token, since anyone who finds the URL could otherwise trigger builds.

Polling: the CI server asks

When the Git server cannot reach the CI server (a firewall, a laptop, a private network), the CI server has to ask. Polling compares the remote's current commit with the one it built last time, using git ls-remote, which reads refs without downloading any objects. Jenkins calls this "Poll SCM" and takes a cron-like schedule such as H/5 * * * * (every five minutes, with a hashed offset to spread load).

Bash Polling with git ls-remote

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first poll found a commit it had never built and started a build; the second saw the same hash and did nothing. After the developer pushed, the third poll noticed the new hash. The polling interval is the delay people wait before a build starts, and the cost is one cheap request per interval per job. Prefer webhooks, and keep polling as a fallback.

What a pipeline checkout really does

When a Jenkins pipeline runs checkout scm (or the shorthand git url: ...), or a GitLab runner starts a job, the same sequence happens in a workspace folder on the build agent:

  1. If the folder has no repository, git init and add the remote (or git clone).
  2. git fetch the branches or refspecs needed, often with --depth for speed.
  3. Check out a specific commit, usually in a detached HEAD state, because the server wants to build exactly the commit that triggered the run and not "whatever the branch points to a second later".
  4. Optionally update submodules, fetch tags, pull LFS files, and clean the tree.

Here is that sequence with plain Git: a shallow clone, a pinned commit and what git status says afterwards.

Bash A shallow, pinned checkout like a CI agent does

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The shallow workspace had one commit and reported true for --is-shallow-repository. Checking out the trigger hash produced HEAD detached at ..., which is normal on CI and not an error. git describe --tags failed in the shallow clone because the tag's commit is not there, so a build script that derives its version number from tags needs git fetch --unshallow --tags (in Jenkins, the "Advanced clone behaviours" option or depth: 0). After that, describe printed v1.0-2-g..., meaning two commits after the tag. See Tags for how describe works and Remotes and Transport for shallow clones.

If a build needs the branch name, take it from the CI server's variables (BRANCH_NAME, GIT_BRANCH, CI_COMMIT_REF_NAME) instead of git branch --show-current, which prints nothing in a detached HEAD.

Workspaces: reuse or start fresh

Cloning a big repository for every build is slow, so most servers reuse the workspace between runs: fetch the new commits, move to the target commit, and clean up. The cleanup is what keeps builds reproducible. A leftover file from the previous build can make a broken build pass.

Bash Resetting a reused workspace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

git reset --hard undoes edits to tracked files and git clean -ffdx removes untracked files, untracked directories (-d), ignored files (-x) and even nested repositories (the doubled -f). Jenkins offers the same as "Clean before checkout" and "Clean after checkout". Read git clean in Depth first so you know exactly what -x will delete on your agent, because a build cache you wanted to keep lives in ignored folders. A common compromise is to keep dependency caches outside the workspace.

Alternatives, in rising order of strictness: do nothing (fast, risky), git clean -fd (keeps ignored caches), git clean -ffdx (reproducible), or a brand-new container or folder per build (slowest, most reliable). Ephemeral agents (containers, worktrees) get this for free.

Jenkins configuration, read-only

A declarative pipeline keeps the build definition in a Jenkinsfile at the root of the repository, so it is versioned with the code and reviewed in pull requests. The Git-related options live in the options and checkout steps:

GROOVY
pipeline {
  agent any
  options {
    skipDefaultCheckout(true)
    timeout(time: 20, unit: 'MINUTES')
  }
  triggers { pollSCM('H/5 * * * *') }
  stages {
    stage('Checkout') {
      steps {
        checkout([$class: 'GitSCM',
          branches: [[name: '*/main']],
          extensions: [
            [$class: 'CloneOption', shallow: true, depth: 1, noTags: false],
            [$class: 'CleanBeforeCheckout']
          ],
          userRemoteConfigs: [[url: '[email protected]:team/app.git', credentialsId: 'deploy-key']]
        ])
      }
    }
    stage('Test') {
      steps { sh './run-tests.sh' }
    }
  }
}

For several branches and pull requests, a multibranch pipeline scans the repository, creates a job for every branch that has a Jenkinsfile and removes jobs when the branch is deleted. GitLab CI uses .gitlab-ci.yml with variables like GIT_DEPTH and GIT_STRATEGY (fetch or clone), and Buildkite and others have comparable settings. The concepts are the same, only the names differ.

Credentials and security

A CI agent that can read your private repository holds a secret. Keep these rules:

  • Use a read-only deploy key or a token with minimal scope, one per repository, rather than a developer's personal account. Store it in the CI server's credentials store (Jenkins credentialsId), never in the Jenkinsfile or the repository.
  • Prefer SSH keys or tokens over passwords, and never put a token in a remote URL: it appears in logs and in .git/config. See Setup and Config for credential helpers.
  • Treat pull requests from forks as untrusted code. Do not give those builds access to secrets or deployment keys. Most systems let you choose who may trigger builds.
  • Pin what you build: use the commit hash from the trigger, not the moving branch name, and check signatures of tags for releases.
  • Report back: set a commit status (success, failure) through the hosting API so pull requests show the result, then require it with branch protection, as described in Pull Requests.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main difference between a webhook trigger and polling?

    Webhooks push a notification when something changes. Polling checks on a schedule, for example with git ls-remote, which costs a request even when nothing changed.
  2. 2. Which command lets a CI server learn the current commit of a branch without downloading objects?

    git ls-remote reads the ref advertisement from the server and prints hashes and ref names only.
  3. 3. A build script runs git describe --tags and fails on CI, but works on your laptop. What is the likely cause?

    A depth-1 clone lacks older commits and the tags that point to them. Fetch tags and unshallow, or configure the checkout to do so.
  4. 4. Why is a detached HEAD common in CI checkouts?

    Building a pinned hash makes the result reproducible even if new commits arrive during the run.
  5. 5. Which pair of commands brings a reused workspace back to a pristine state, including ignored build output?

    reset --hard restores tracked files; clean -ffdx removes untracked, ignored and nested-repository content too.

Summary

  • CI servers learn about new commits through webhooks (instant, push) or polling with git ls-remote (scheduled, pull); the commit hash identifies what to build and where to report status.
  • A pipeline checkout fetches, often shallowly, and checks out the exact triggering commit as a detached HEAD.
  • Shallow clones lack history and tags: unshallow or fetch tags when a script needs git describe or git log depth.
  • Reused workspaces need git reset --hard plus git clean; use -x only when you do not need ignored caches.
  • Keep the pipeline definition in the repository (Jenkinsfile, .gitlab-ci.yml), use minimal read-only credentials stored in the CI server, and treat fork pull requests as untrusted.

Next: Deploying with Git