Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 40 ยท Projects

Project: A Monorepo Pipeline with Changed-Path Builds

Build a Jenkins pipeline for a monorepo that only builds and tests the services whose files changed, runs them in parallel, reuses a shared library and caches dependencies.

A monorepo keeps many services and libraries in one Git repository. It makes sharing code easy, but it creates a CI problem: if every push rebuilds and retests everything, a one-line change to the docs costs forty minutes of compute. The goal of this project is a pipeline that asks "what changed?" and then builds only what that change can affect.

You will design the repository layout, write a declarative Jenkinsfile with changeset conditions and parallel stages, move the repeated parts into a shared library, add dependency caching, and finally model the whole decision with a runnable JavaScript stand-in. The real pipeline needs a Jenkins server, so its code is read-only. The TryIt blocks are stand-ins that reproduce the logic faithfully, and the lesson says where they differ.

The monorepo problem

Build everything Simple, but slow and wasteful.
Build what changed Fast, but you must map files to services.
Shared code A change in libs/ affects every user of it.

Changed-path builds are a mapping from files to stages.

1 / 2

The repository layout

Assume this layout. Each service has its own folder and its own build command; libs/shared is used by both services.

TEXT
repo/
  Jenkinsfile
  services/
    api/        (Java, built with Gradle)
    web/        (Node, built with npm)
  libs/
    shared/     (used by api and web)
  docs/

The rules we want are:

Changed file What must run
services/api/** api build and tests
services/web/** web build and tests
libs/shared/** both api and web (they depend on it)
docs/** only nothing heavy
Jenkinsfile itself everything (the pipeline changed)

The last row is easy to forget. If the Jenkinsfile changes, any previous "nothing changed" conclusion may be wrong, so the safe move is to build all.

Choosing stages with changeset

Declarative pipelines have a when directive with a changeset condition. It matches Ant-style patterns against the files changed in the build. Each service gets its own stage guarded by it, and a stage can list several patterns with anyOf so that shared code triggers both services.

GROOVY
pipeline {
  agent none
  options {
    timeout(time: 30, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '30'))
  }
  stages {
    stage('Services') {
      parallel {
        stage('api') {
          when {
            beforeAgent true
            anyOf {
              changeset 'services/api/**'
              changeset 'libs/shared/**'
              changeset 'Jenkinsfile'
            }
          }
          agent { label 'linux && java21' }
          steps {
            dir('services/api') {
              sh './gradlew --no-daemon test assemble'
            }
          }
          post {
            always { junit 'services/api/build/test-results/**/*.xml' }
          }
        }
        stage('web') {
          when {
            beforeAgent true
            anyOf {
              changeset 'services/web/**'
              changeset 'libs/shared/**'
              changeset 'Jenkinsfile'
            }
          }
          agent { label 'linux && node22' }
          steps {
            dir('services/web') {
              sh 'npm ci && npm test && npm run build'
            }
          }
        }
      }
    }
  }
}

Three details deserve attention:

  • beforeAgent true makes Jenkins evaluate when before it reserves an agent. Without it, a skipped stage would still wait in the queue for a free executor and then do nothing.
  • agent none at the top means no executor is held while the pipeline decides. Each stage asks for the agent it needs.
  • parallel runs the selected stages at the same time. Stages that are skipped by when simply show as skipped.

changeset looks at the changes Jenkins recorded for this build, which normally means the commits since the previous build of the same branch. On the very first build of a branch there is no previous build, so the list can be empty and every changeset condition false. Teams handle this by also building everything on the main branch and on a nightly schedule, so a missed change never hides for long. On pull requests, comparing against the target branch is more reliable; see below.

A more reliable alternative: diff against the target branch

For pull requests you usually want "everything this PR changes compared with the target branch", not "since the last build". You can compute that yourself and use an expression:

GROOVY
stage('Detect changes') {
  agent { label 'linux' }
  steps {
    script {
      def base = env.CHANGE_TARGET ?: 'main'
      sh "git fetch --no-tags origin ${base}"
      def out = sh(returnStdout: true,
                   script: "git diff --name-only origin/${base}...HEAD").trim()
      env.CHANGED = out
    }
  }
}

CHANGE_TARGET is set by multibranch pipelines for pull requests (see Multibranch Pipelines and Pull Requests). Later stages then use when { expression { env.CHANGED.contains('services/api/') } }. This is more code, but it works the same on the first build, on reruns and on pull requests.

Removing repetition with a shared library

Two nearly identical stages become six when you add more services. Move the repeated shape into a shared library step so each Jenkinsfile only says what, not how (see Shared Libraries). A global step in the library's vars/buildService.groovy might look like this:

GROOVY
// vars/buildService.groovy
def call(Map cfg) {
  dir(cfg.path) {
    sh cfg.command
  }
  if (cfg.reports) {
    junit cfg.reports
  }
}

And the Jenkinsfile becomes:

GROOVY
@Library('ci-lib@v2') _

// ...inside the api stage:
steps {
  buildService(path: 'services/api',
               command: './gradlew --no-daemon test assemble',
               reports: 'services/api/build/test-results/**/*.xml')
}

Pinning the library with @v2 means a change to the library cannot silently break every pipeline at once.

Caching dependencies

Downloading dependencies is often the slowest part of a build. A cache is correct only if its key changes whenever the inputs change. For Node, the inputs are package-lock.json; for Gradle, the lockfile or build files. The idea is the same everywhere: hash the lockfile, and restore a cache stored under that hash if one exists.

On a persistent agent, a simple way is to keep a cache directory on the agent and name it after the hash:

GROOVY
steps {
  dir('services/web') {
    script {
      def key = sh(returnStdout: true, script: 'sha256sum package-lock.json | cut -c1-12').trim()
      env.CACHE_DIR = "/var/cache/ci/web-${key}"
    }
    sh 'mkdir -p "$CACHE_DIR" && npm ci --cache "$CACHE_DIR"'
  }
}

Note the single quotes around the second script: the shell, not Groovy, expands $CACHE_DIR. With ephemeral agents (containers that vanish), you need a plugin or a registry to carry the cache between builds; that is outside this lesson, but the key idea does not change.

A stand-in: pick stages from changed files

The first TryIt models the decision table. Patterns use ** (any depth) and * (one folder level). The function select returns which services to build and why. It is a simplified glob matcher, not Jenkins' real implementation, but it shows the same behavior: shared code pulls in dependents, and a changed Jenkinsfile builds everything.

JS Which services need a build?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the last scenario carefully: with no known changes, nothing builds. That is the "first build" trap from the warning above, shown as data. A real pipeline would add a rule such as "on the main branch, build everything".

A stand-in: cache keys and parallel timing

The second TryIt does two things. It computes a cache key from lockfile text, so you can see that unrelated edits keep the key and a dependency change breaks it. It also estimates the stage time: with parallel stages, the wall-clock time is the longest selected service, not the sum. The hash is a small deterministic function, not the SHA-256 a real build would use.

JS Cache keys and parallel build time

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The dependency bump produces a new key and a full download, while the code-only change reuses the cache. That is exactly the property you want from a cache key.

Putting it together

A good monorepo pipeline combines these pieces:

  1. Detect what changed (changeset, or a diff against the target branch).
  2. Select stages with when and beforeAgent true.
  3. Run in parallel on appropriately labeled agents.
  4. Reuse build logic through a pinned shared library.
  5. Cache dependencies with a content-based key.
  6. Safety net: build everything on main and nightly, and when the Jenkinsfile changes.

Before you trust a changed-path pipeline, deliberately break something in libs/shared on a branch and confirm both services build. The mapping is only as good as your dependency table, and the table goes stale as the code evolves.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does beforeAgent true do inside a when block?

    Without it Jenkins allocates the agent first, which wastes queue time for a stage that will be skipped.
  2. 2. A change in libs/shared should trigger which stages in our layout?

    Shared code affects its dependents, so the patterns for each dependent service include libs/shared/**.
  3. 3. Why can changeset miss changes on the first build of a branch?

    changeset relies on changes recorded for the build. Build everything on main and nightly, or diff against the target branch.
  4. 4. Two services take 240s and 130s. They run in parallel. About how long is the stage?

    Parallel stages take as long as the slowest branch: 240 seconds, plus a little overhead.
  5. 5. What makes a dependency cache key correct?

    A content-based key means the cache is reused when inputs are identical and rebuilt when they are not.

Summary

  • A monorepo pipeline maps changed files to the stages that need to run.
  • Use when { changeset ... } with anyOf and beforeAgent true, or diff against the target branch for pull requests.
  • Shared code must trigger every dependent service, and a changed Jenkinsfile should build everything.
  • Run selected services in parallel, keep stage logic in a pinned shared library, and cache with content-based keys.
  • Always keep a safety net: full builds on main and on a schedule.

Next, build a small pipeline engine yourself in Project: Build a Mini Jenkins Pipeline Engine.