Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 44 ยท Performance and Operations

Making Pipelines Fast

Speed up CI/CD pipelines with caching, parallel jobs, test sharding, path filters, shallow clones, small images and critical path analysis.

A pipeline that takes four minutes is part of your work rhythm. A pipeline that takes forty minutes is something people avoid: they batch up changes, switch to other tasks while waiting, and merge without looking at results. Slow pipelines are not just annoying, they make every other practice in this course less effective, because fast feedback is the whole point of CI. In What Is CI/CD? you saw that integration is cheap only when it is frequent, and frequency depends on speed.

The good news is that most slow pipelines are slow for a handful of repeatable reasons: work is repeated, work is done in sequence that could run side by side, work is done that nobody needed, or too much data is moved around. This lesson walks through each cause and ends with a way to find where your time really goes: critical path analysis. The YAML below is read-only; the runnable stand-ins calculate and compare timings, which is the thinking you need before touching a config file.

Measure before you optimize

Never start by guessing. Open a recent run and look at three numbers:

  1. Wall-clock time: how long a developer waits, from push to final result.
  2. Per-job and per-step time: where those minutes go.
  3. Queue time: how long jobs waited for a runner before starting. If this is large, no amount of tuning inside the job helps; you need more runners (see Self-Hosted Runners and Runner Costs).

Wall-clock time is not the sum of all job times, because jobs run in parallel. This is the most important idea in the lesson, and we will return to it with the critical path.

Do less work

The cheapest step is the one that never runs.

Path filters. If only the documentation changed, there is no reason to run the end-to-end suite. Use paths or paths-ignore on the trigger:

YAML
on:
  pull_request:
    paths-ignore:
      - "docs/**"
      - "**.md"

One caution: if a workflow is a required status check and it is skipped by a path filter, the check stays "pending" and can block merging. Teams handle this either with a tiny always-running "gate" job or with a path-detection step inside the workflow. The Monorepos lesson goes deeper.

Cancel superseded runs. When a developer pushes twice in a minute, the first run is already obsolete:

YAML
concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Use this for tests, not for deployments.

Shallow clones. By default actions/checkout fetches a single commit (depth 1), which is fast. Do not turn on fetch-depth: 0 (full history) unless a step truly needs it, such as changelog generation or "what changed since the last tag" logic. On a large repository the difference can be minutes.

Skip the unnecessary. Don't install tools you never use, don't build documentation on every commit, and don't run the slow nightly-grade checks on every pull request; move them to a schedule trigger.

Reuse work with caching

Downloading and compiling the same dependencies on every run is the largest avoidable cost in many pipelines. The Caching and Artifacts lesson covers the mechanics. For performance, remember four rules:

  • Key the cache on the lock file, so it changes only when dependencies change.
  • Add a restore-keys prefix so a near match is better than nothing.
  • Cache downloads (the package manager's cache), not always the installed folder. It is safer and usually fast enough.
  • Measure the benefit. Restoring a huge cache can take longer than downloading a small dependency set. A cache that saves 10 seconds and costs 40 to restore is a loss.
YAML
- uses: actions/setup-node@v4
  with:
    node-version: 22
    cache: npm          # keys on package-lock.json automatically
- run: npm ci

Docker builds have their own layer cache. Order your Dockerfile so rarely changing steps (installing dependencies) come before often changing ones (copying source code), and use the registry or GitHub cache backend with BuildKit, as shown in Building and Pushing Docker Images.

Do more at once

CI machines are cheap compared to people waiting. If two pieces of work do not depend on each other, run them in parallel.

Split jobs. Lint, unit tests and a type check rarely depend on each other. Put them in separate jobs with no needs, and they run side by side. Use needs only where one job truly requires another's output, as covered in Jobs, Steps and Dependencies.

Shard slow test suites. One 20-minute suite can become four 5-minute shards using a matrix. Each shard runs a slice of the tests:

YAML
jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test --shard=${{ matrix.shard }}/4

Many test runners have a built-in shard option like this one. Good sharding balances by duration, not by file count, because ten tiny files and one huge file are not an even split. Some tools record past test timings and use them to balance shards.

Be aware of the trade-off: every parallel job pays its own startup cost (provisioning a runner, checkout, restoring the cache, installing). Four shards of 5 minutes do not take 5 minutes if each has 1.5 minutes of setup; they take 6.5. Sharding pays off for long jobs and hurts short ones.

Move less data

  • Smaller images. A 1.5 GB build image takes much longer to pull than a 150 MB one. Prefer slim or minimal base images, and use multi-stage builds so compilers and build tools stay out of the final image.
  • Prebuilt tool images. If every run spends two minutes installing the same ten tools, bake them into a custom image (used with container: or as a self-hosted runner image) and pull it once.
  • Targeted artifacts. Upload only what a later job needs, not the whole workspace. Large artifacts cost upload and download time and storage.
  • Compress and exclude. Exclude node_modules or build caches from artifacts and rebuild or restore them where needed.

Critical path analysis

Imagine this pipeline:

flowchart LR
    L["lint: 1 min"] --> D
    U["unit: 4 min"] --> K
    B["build: 3 min"] --> E["e2e: 8 min"]
    B --> K["docker: 2 min"]
    E --> D["deploy: 1 min"]
    K --> D

Each box is a job with its duration; arrows show needs. Lint and unit start immediately, and build starts alongside them.

The jobs add up to 1 + 4 + 3 + 8 + 2 + 1 = 19 minutes of work, but the pipeline does not take 19 minutes, because lint, unit and build run at the same time. The pipeline takes as long as its critical path: the longest chain of dependent jobs from start to finish. Here the chain build, e2e, deploy takes 3 + 8 + 1 = 12 minutes. Speeding up lint or unit tests changes nothing about wall-clock time. Speeding up e2e saves minutes immediately.

That is why measuring matters. Optimizing a job off the critical path feels productive and achieves nothing; optimizing one on the path pays off until another path becomes the longest.

Stand-in: critical path calculator

This stand-in computes the earliest finish time of each job (its own duration plus the latest finish among its dependencies) and then walks back to find the critical path. Then it splits the end-to-end job into four shards to show how the bottleneck moves. It works on plain data, just like a real scheduler.

JS Find the critical path of a job graph

Press Run (or Ctrl + Enter) to see the console output here.

No output.

After sharding, the end-to-end work shrinks from 8 minutes to 2.5, wall-clock time drops from 12 to 7 minutes, and the critical path switches to a different chain: unit, docker, deploy. The total work actually grew (21 minutes instead of 19, because every shard pays setup), yet the developer waits 5 minutes less. That is typical: you remove one bottleneck and the next one appears. Stop when the next gain is smaller than the effort.

Stand-in: is sharding worth it?

Setup cost decides whether parallelism helps. This second stand-in calculates the wall time of a job split into different numbers of shards when each shard pays a fixed setup cost.

JS Shard count versus setup overhead

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that going from 8 to 16 shards saves only 1.25 minutes of waiting but adds 12 billed machine minutes (32 to 44). Choose the point where the curve flattens.

A faster pipeline checklist

  • Find the critical path; optimize only jobs on it.
  • Skip work with path filters, concurrency cancellation and shallow clones.
  • Cache dependencies by lock file, and measure that the cache actually helps.
  • Run independent jobs in parallel; shard long test suites, balanced by duration.
  • Use small, prebuilt images and upload only the artifacts you need.
  • Watch queue time and flaky retries; both quietly add minutes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A pipeline has jobs that total 30 minutes of work, but several run in parallel. What determines the wall-clock time?

    Parallel jobs overlap, so the pipeline finishes when the longest dependent chain finishes.
  2. 2. You make a job faster, but it is not on the critical path. What happens to the pipeline's wall-clock time?

    Only jobs on the critical path set the total time. Speeding up another job saves resources but not waiting time.
  3. 3. Why can too many test shards be a bad idea?

    Every shard pays for runner startup, checkout and dependency installation. Past a point you add cost without saving much time.
  4. 4. Which change is safe for pull request tests but dangerous for deployment workflows?

    Cancelling an obsolete test run is harmless, but cancelling a deployment halfway can leave a system in a partial state.
  5. 5. When should you use fetch-depth: 0 on checkout?

    The default shallow checkout is fast. Full history is needed only by steps that inspect past commits or tags.

Summary

  • Measure first: wall-clock time, per-step time and queue time.
  • Wall-clock time equals the critical path, the longest chain of dependent jobs, not the sum of job times.
  • Do less: path filters, cancelling superseded runs, shallow clones.
  • Reuse work: lock-file cache keys, restore-keys, Docker layer caching; verify the cache is a net win.
  • Do more at once: split independent jobs and shard long suites, but remember each shard pays setup cost.
  • Move less data: slim images, prebuilt tool images, targeted artifacts.

Next, you will apply these ideas to repositories that hold many projects: CI for Monorepos.