Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 36 ยท GitLab CI/CD Pipelines

Needs, Includes and Reusable Pipelines

Speed up and organize GitLab pipelines: DAG scheduling with needs, include for shared templates, parallel matrix jobs, child pipelines and multi-project triggers.

Stages are simple but blunt: a stage waits for every job of the previous stage, even those it has nothing to do with. As pipelines grow, you also end up with a .gitlab-ci.yml that is hundreds of lines long and copied between projects. This lesson closes the chapter with the tools that fix both problems: needs for faster ordering, and include, parallel:matrix, child pipelines and triggers for organization and reuse.

needs: a graph instead of a staircase

Add needs: to a job to say exactly which jobs it depends on. GitLab then starts the job as soon as those jobs finish, ignoring stage boundaries. Stages still control display and default ordering, but the execution becomes a directed acyclic graph (DAG).

YAML
stages: [build, test, deploy]

build-backend:
  stage: build
  script: ./build-backend.sh

build-frontend:
  stage: build
  script: ./build-frontend.sh

test-backend:
  stage: test
  needs: [build-backend]
  script: ./test-backend.sh

test-frontend:
  stage: test
  needs: [build-frontend]
  script: ./test-frontend.sh

deploy:
  stage: deploy
  needs: [test-backend, test-frontend]
  script: ./deploy.sh

Here test-backend starts the moment build-backend ends, even if build-frontend is still running. A job with needs also downloads artifacts only from the listed jobs (use artifacts: false inside a need to skip that). A job with needs: [] has no dependencies and starts immediately, whatever its stage.

flowchart LR
  BB["build-backend"] --> TB["test-backend"]
  BF["build-frontend"] --> TF["test-frontend"]
  TB --> D["deploy"]
  TF --> D

The two columns of work no longer wait for each other.

The stand-in below computes how long a pipeline takes in both styles, so you can see the saving. Durations are in minutes and invented.

Python Stand-in: stages versus needs timing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With stages the pipeline waits for the slowest job of every stage: 6 + 5 + 1 = 12 minutes. With needs only the longest chain counts: the backend chain takes 6 + 2 = 8 minutes, the frontend chain 2 + 5 = 7, so deploy finishes at minute 9. The speedup appears whenever the slowest job of one stage is not the slowest of the next. Change the durations to find cases where both styles tie.

include: share and split configuration

The include keyword pulls in other YAML files and merges them as if they were written in place. Four sources:

YAML
include:
  - local: .gitlab/ci/build.yml                 # a file in this repo
  - project: acme/ci-templates                  # a file in another project
    ref: v2.1.0
    file: /templates/node.yml
  - remote: https://example.com/shared-ci.yml   # a public URL
  - template: Jobs/SAST.gitlab-ci.yml           # a template shipped with GitLab

Practical uses:

  • Split a huge file into build.yml, test.yml and deploy.yml in a .gitlab/ci/ folder.
  • Create a company-wide templates project and include it everywhere, pinning ref to a tag so a change in the template cannot break every project at once.
  • Turn on GitLab's built-in scanners with a one-line template: include (see the security chapter).

Included files can contain hidden jobs and jobs; your own file can override them by redefining the same key (keys merge, with your local file winning), and extends makes the reuse explicit.

For parameterized reuse, newer GitLab versions offer CI/CD components and spec:inputs, which let a template accept typed arguments, for example:

YAML
spec:
  inputs:
    node_version:
      default: "22"
---
test:
  image: node:$[[ inputs.node_version ]]
  script: npm test

and you pass inputs: when including it. Check the current documentation for the include syntax of components, since this area changes quickly.

parallel:matrix: one job, many combinations

matrix expands a job into one copy per combination of variable values:

YAML
test:
  image: python:$PYTHON
  script:
    - pip install -r requirements.txt
    - pytest
  parallel:
    matrix:
      - PYTHON: ["3.11", "3.12", "3.13"]
        OS_FLAVOR: ["slim", "alpine"]

That creates six jobs. Each run gets its own values of PYTHON and OS_FLAVOR, and the job names show them. A downstream job can need the matrix by listing the same variables.

Python Stand-in: expand a matrix into jobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Child pipelines and multi-project pipelines

Sometimes one flat file is not enough.

  • A parent-child pipeline starts another pipeline inside the same project using trigger:include. It suits monorepos (one child per component) and dynamic pipelines, where an earlier job generates a YAML file and an artifact passes it on:
YAML
frontend:
  trigger:
    include: frontend/.gitlab-ci.yml
    strategy: depend     # the parent waits and mirrors the child's status
  rules:
    - changes: [frontend/**/*]
  • A multi-project pipeline triggers a pipeline in another project, for example a deployment project:
YAML
deploy-infra:
  stage: deploy
  trigger:
    project: acme/infrastructure
    branch: main
    strategy: depend

Child pipelines have their own jobs and graph, and keep each file small. The rules:changes condition above runs a child only when its folder changed, which makes monorepos fast.

Putting a bigger pipeline together

A project with several components might use:

  1. workflow:rules to avoid duplicate pipelines.
  2. include for shared templates and scanners.
  3. Stages for readability, and needs for speed.
  4. rules:changes and child pipelines per component.
  5. A manual deploy-production job with resource_group and a protected environment (see the next chapter on runners and jobs and the later one on environments).

Common mistakes

  • Cyclic needs: A needs B and B needs A. GitLab rejects the pipeline.
  • Needing a job that does not exist in this pipeline (excluded by rules): use optional: true inside the need.
  • Unpinned includes: an included file changes and silently breaks all projects; pin to a tag or commit.
  • Too many tiny child pipelines: each has overhead; use them where isolation pays off.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does needs change about job execution?

    needs builds a dependency graph so jobs do not have to wait for the whole previous stage.
  2. 2. Why pin a project include to a tag with ref?

    Pinning gives you control over when a shared template change reaches your pipeline.
  3. 3. How many jobs does a matrix of 3 versions and 2 flavors create?

    A matrix creates one job per combination: 3 times 2 equals 6.
  4. 4. Which keyword starts a pipeline in the same project from another YAML file?

    trigger:include starts a child pipeline from the given file.

Summary

  • needs turns stages into a dependency graph so independent work runs sooner.
  • include (local, project, remote, template) splits and shares configuration; pin versions of shared templates.
  • parallel:matrix expands one job into many combinations.
  • Child and multi-project pipelines organize monorepos and cross-project deployment.
  • Avoid cycles, unpinned includes and needless complexity.

Next: Runners: Where Your Jobs Run