Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 44 ยท Performance and Operations

CI for Monorepos

Run CI only for what changed in a monorepo: path filters, affected-package detection, dynamic matrices, per-package versions and GitLab child pipelines.

A monorepo is a single repository that holds many projects: a web app, an API, shared libraries, maybe documentation. Teams choose it because shared code is easy to change atomically (one commit updates a library and all its users) and because tooling, rules and history live in one place. The cost shows up in CI. A naive pipeline that builds and tests everything on every commit gets slower with every package you add, until a one-line change to a README triggers an hour of work.

The remedy is to make the pipeline change-aware: do the work for the parts of the repository that were affected, and skip the rest. This lesson builds that up in layers, from simple path filters to computing affected packages from a dependency graph and feeding the result to a dynamic matrix. As usual, the YAML is read-only. The runnable stand-in implements the heart of it, the "what is affected?" calculation, which is plain logic you can reason about.

Layout and the problem

A typical JavaScript monorepo might look like this:

TEXT
packages/
  core/      (no dependencies)
  utils/     (no dependencies)
  ui/        (depends on core, utils)
  api/       (depends on core)
  web/       (depends on ui, api)
docs/
package.json
package-lock.json

If someone edits packages/utils/src/format.js, which packages need new tests? Obviously utils. But ui imports utils, and web imports ui, so a bug introduced in utils could break both. They are affected by the change even though none of their files changed. core and api are not affected and can be skipped. Working this out reliably is the central problem of monorepo CI.

Level 1: path filters

The simplest tool is the paths filter on the trigger, which you met in Triggers. Give each package its own workflow that runs only when its files (or its dependencies' files) change:

YAML
# .github/workflows/web.yml
name: web

on:
  pull_request:
    paths:
      - "packages/web/**"
      - "packages/ui/**"
      - "packages/api/**"
      - "packages/core/**"
      - "packages/utils/**"
      - "package-lock.json"
      - ".github/workflows/web.yml"

This works, and it is easy to understand, but notice the weakness: the list of dependencies is written by hand in every workflow. When web starts depending on a new package, someone must remember to update the filter. Forgotten entries mean skipped tests that should have run, which is the worst kind of bug in CI because a green check lies. Also remember the problem from the previous lesson: a workflow skipped by a path filter never reports a status, so it cannot be used directly as a required check. Path filters are fine for small repositories with a few independent projects.

Level 2: detect affected packages

The scalable approach is a first job that computes what is affected, then later jobs act on that list. The steps:

  1. Get the list of changed files (compare the pull request with its base branch).
  2. Map each file to the package that owns it. Files that belong to no package, but influence everything (the lock file, the CI config, shared tooling), mark everything as affected.
  3. Walk the dependency graph backwards to add every package that depends, directly or indirectly, on a changed package.
flowchart BT
    core["core"] --> ui["ui"]
    utils["utils (changed)"] --> ui
    core --> api["api"]
    ui --> web["web"]
    api --> web

Arrows point from a dependency to the package using it. A change in utils affects ui, then web.

Many tools do this for you: Nx (nx affected), Turborepo (--filter=...[origin/main]), Bazel, Pants, Lerna and workspace features of package managers. Learning the idea first helps you configure any of them, and a small script is often enough for a modest repository.

Stand-in: compute the affected set

This stand-in implements steps 2 and 3 on plain data. It maps changed paths to packages, treats a few global files as "rebuild everything", and expands through reverse dependencies with a breadth-first walk. The output is the JSON a CI job could pass on to a matrix.

JS Affected packages from changed files

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Look at the "docs only" case. Nothing under packages/ changed, so the list is empty. In a real workflow that must be handled explicitly (more on that below), or GitHub rejects an empty matrix.

Level 3: dynamic matrices

A matrix normally lists its values in the YAML. For a monorepo you want the list to come from the detection job. The pattern uses a job output and fromJSON, covered in Jobs, Steps and Dependencies and Matrix Builds:

YAML
name: ci

on:
  pull_request:

jobs:
  detect:
    runs-on: ubuntu-latest
    outputs:
      packages: ${{ steps.affected.outputs.packages }}
      any: ${{ steps.affected.outputs.any }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0          # we need the base branch to diff against
      - id: affected
        run: |
          node tools/affected.mjs "origin/${{ github.base_ref }}" > affected.json
          echo "packages=$(jq -c .packages affected.json)" >> "$GITHUB_OUTPUT"
          echo "any=$(jq '.packages | length > 0' affected.json)" >> "$GITHUB_OUTPUT"

  test:
    needs: detect
    if: needs.detect.outputs.any == 'true'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        package: ${{ fromJSON(needs.detect.outputs.packages) }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm test --workspace "packages/${{ matrix.package }}"

  ci-ok:
    if: always()
    needs: [detect, test]
    runs-on: ubuntu-latest
    steps:
      - run: |
          if [ "${{ contains(needs.*.result, 'failure') }}" = "true" ] || [ "${{ contains(needs.*.result, 'cancelled') }}" = "true" ]; then
            exit 1
          fi

Study the pieces:

  • fetch-depth: 0 is one of the few justified uses of full history: the script needs the base branch commit to compare against.
  • The detection step writes packages as compact JSON (jq -c) to $GITHUB_OUTPUT. The matrix reads it back with fromJSON.
  • if: needs.detect.outputs.any == 'true' skips the test job when nothing is affected, avoiding the empty-matrix error. (Outputs are strings, so compare with the string 'true'.)
  • The final ci-ok job runs always() and fails if anything failed or was cancelled. Skipped jobs count as fine here. This single job is what you mark as the required status check in branch protection. Because it always runs, it always reports, and it solves the "skipped workflow stays pending" problem.

Make the "what is affected" logic a script in your repository, not inline YAML. A script can be tested, run on your laptop, and reused by GitHub Actions, Jenkins and GitLab alike.

Caching in a monorepo

One lock file at the root means one dependency cache shared by all jobs, which is good. Keep the cache key tied to that lock file. Build tools add a second kind of cache: a task cache that remembers "this package with these inputs already built and tested". Remote caches (Nx Cloud, Turborepo remote cache and others) share task results between CI runs and developer machines, so unchanged packages are not rebuilt even when they are "affected" by your own definition. Treat this as an optimization after you have correct affected detection, not instead of it.

Per-package versioning and releases

Libraries in a monorepo often have independent versions. A change to ui should bump ui and publish it, not publish all packages. The inputs are the same as in single-package releases (see Versioning and Automated Releases), with two additions:

  • Commits are attributed to packages by the files they touch, or by a scope in the message such as feat(ui): add a Tooltip.
  • When a package changes, its dependents may need a version bump too, because their dependency range or lockfile changes. Tools such as Changesets, Lerna and release-please manage this. A common design is to require a small "change file" in every pull request that says which packages change and whether it is a patch, minor or major bump.

Tags help keep versions separate: use names such as ui-v2.4.0 and trigger publish workflows on tags: ["ui-v*"]. You will put this together in the monorepo release project.

GitLab: child pipelines

GitLab has a built-in answer to dynamic work: parent-child pipelines. A parent pipeline starts a child pipeline from a YAML file in the repository, or from a file generated by an earlier job as an artifact. A static version routes by path with rules:changes:

YAML
web-pipeline:
  stage: test
  trigger:
    include: packages/web/.gitlab-ci.yml
    strategy: depend
  rules:
    - changes:
        - packages/web/**/*
        - packages/ui/**/*

strategy: depend makes the parent job wait for the child and take its result. The dynamic version has a generator job that writes child.yml as an artifact and a trigger job that includes it:

YAML
generate:
  stage: build
  script:
    - node tools/make-child-pipeline.mjs > child.yml
  artifacts:
    paths: [child.yml]

run-children:
  stage: test
  needs: [generate]
  trigger:
    include:
      - artifact: child.yml
        job: generate
    strategy: depend

Jenkins can do the same with a when { changeset "packages/web/**" } condition in a declarative stage, and by computing the list in a script step. The idea is identical in all three tools: decide first, then act.

Pitfalls

  • Missing global triggers. Changes to the lock file, shared config, the CI definition and base images must widen the affected set. Include them in your "global files" list.
  • Hidden dependencies. If package web reads a file from docs/ at build time, the graph does not know. Declare such inputs or add them to the filters.
  • Shallow clones. Detection needs the base commit. If the diff fails silently, you may skip everything. Make the script fail loudly (and when in doubt, run everything).
  • Merge queues and main. On pushes to main you often want a full run or a diff against the previous release tag rather than against a pull request base.
  • Required checks. Always have a single always-running summary job, as in ci-ok.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the sample graph, ui depends on utils, and web depends on ui. Only a file in utils changes. Which packages are affected?

    A change affects the package itself and everything that depends on it, directly or indirectly. That is utils, then ui, then web.
  2. 2. Why is a hand-written paths list in every workflow risky in a growing monorepo?

    A stale filter produces a green check without running the tests that mattered. Computing the affected set from the dependency graph avoids that.
  3. 3. What problem does a final job with if: always() that fails when needed jobs failed solve?

    Skipped workflows or jobs may never report a status. A summary job that always runs always reports, so it can be the one required check.
  4. 4. What does strategy: depend do on a GitLab trigger job?

    Without it the trigger job succeeds as soon as the child is created. With depend the parent reflects the child's final status.
  5. 5. Why does the detection job check out with fetch-depth: 0?

    The script compares the current commit with the base branch, which needs more than the single latest commit.

Summary

  • A monorepo needs a change-aware pipeline, or CI time grows with every package.
  • Path filters are simple but hand-maintained; they also leave required checks pending when skipped.
  • Affected detection maps changed files to packages and walks the dependency graph backwards; global files such as the lock file affect everything.
  • A detection job can emit JSON that a dynamic matrix reads with fromJSON; guard against an empty list and use one always-running summary job as the required check.
  • Per-package versions use scoped commits or change files and per-package tags.
  • GitLab offers parent-child pipelines (static or generated); Jenkins uses changeset conditions. The principle is the same: decide first, then act.

Next, you will learn what to do when a pipeline goes red: Debugging Failing Pipelines.