Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 44 ยท GitLab CI/CD

Rules, needs, Artifacts and Cache

Control GitLab CI jobs with rules and workflow:rules, speed pipelines up with the needs DAG, pass files with artifacts and reports, and reuse dependencies with cache.

The basic pipeline runs every job on every push, one stage at a time, with each job in a blank environment. Real projects need more: deploy only from the main branch, run extra checks only in merge requests, start a test the moment its build is done instead of waiting for a whole stage, hand compiled files to the next job, and reuse downloaded dependencies.

Four keywords cover that: rules, needs, artifacts and cache. This lesson explains each and the mistakes that go with them.

GitLab cannot run here, so all YAML is read-only (use the CI Lint tool in your project to validate it). The TryIts are JavaScript stand-ins: one evaluates rules the way you read them, the other compares stage-based and needs-based timing. They are teaching models, not GitLab's engine.

rules: when does a job exist?

rules is a list evaluated top to bottom for each job when the pipeline is created. The first rule that matches decides what happens; if none matches, the job is not added to the pipeline.

YAML
deploy-production:
  stage: deploy
  script:
    - ./deploy.sh production
  rules:
    - if: $CI_COMMIT_TAG
      when: manual
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: never

Each rule can have these parts:

  • if: an expression over variables, using ==, !=, =~ (regex match), !~, &&, || and parentheses.
  • changes: matches when listed files changed (for example changes: [src/**/*]).
  • exists: matches when a file exists in the repository.
  • when: on_success (default), manual, delayed, always or never.
  • allow_failure and variables: adjust the job when this rule matches.

A rule with no if, changes or exists always matches, so a final - when: never or a final plain rule serves as the default case.

only and except: the old way

You will meet only: and except: in older files. They are still understood but are no longer developed, and GitLab recommends rules. You cannot combine only/except and rules in one job. In new files, use rules only.

The predefined variables you use most

Variable Meaning
CI_PIPELINE_SOURCE Why the pipeline exists: push, merge_request_event, schedule, web, api, trigger, pipeline and others
CI_COMMIT_BRANCH Branch name. Not set in merge request pipelines and tag pipelines
CI_COMMIT_TAG Tag name, only in tag pipelines
CI_DEFAULT_BRANCH Name of the default branch, usually main
CI_MERGE_REQUEST_ID Set in merge request pipelines
CI_OPEN_MERGE_REQUESTS Set on branch pipelines when the branch has an open merge request

Try it: evaluate rules against pipeline variables

This stand-in parses the common shape of an if expression ($VAR, comparison with string or regex, &&, ||) and walks a job's rules to tell you whether the job is added, and with what when. It is deliberately small: no parentheses, no changes or exists.

JS A tiny rules evaluator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the table the output draws: deploy-staging appears only on the main branch push, release only for a tag that looks like a version, and unit-tests runs for both merge requests and branch pushes. Which brings up the classic trap in the next section.

workflow:rules and merge request pipelines

rules on jobs decide which jobs are in a pipeline. workflow:rules decides whether the pipeline exists at all. A common problem: a branch with an open merge request triggers a branch pipeline on each push and also a merge request pipeline, so every commit runs twice. This standard pattern prevents the duplicate:

YAML
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

Read it as: use the merge request pipeline when it exists; skip the branch pipeline if an MR is open; otherwise run a normal branch pipeline. Merge request pipelines can also test the merged result and make the pipeline status show in the merge request widget. Remember that CI_COMMIT_BRANCH is empty in them, so jobs that should run in MRs need the merge_request_event condition.

needs: a directed acyclic graph

Stages create a strict barrier: nothing in test starts until all of build is done. That is wasteful when a fast job sits behind a slow one that it does not depend on. The needs keyword lets a job start as soon as the jobs it names have finished, regardless of stage:

YAML
build-api:
  stage: build
  script: [./build-api.sh]

build-web:
  stage: build
  script: [./build-web.sh]

test-api:
  stage: test
  needs: [build-api]
  script: [./test-api.sh]

test-web:
  stage: test
  needs: [build-web]
  script: [./test-web.sh]

deploy:
  stage: deploy
  needs: [test-api, test-web]
  script: [./deploy.sh]

The jobs now form a directed acyclic graph (DAG): arrows go from a job to those that need it, with no cycles. needs: [] (an empty list) lets a job start immediately at the beginning. Stages still exist for display and for jobs that have no needs. By default a job with needs also downloads the artifacts of the jobs it needs.

flowchart LR
    BA["build-api"] --> TA["test-api"]
    BW["build-web"] --> TW["test-web"]
    TA --> D["deploy"]
    TW --> D

With needs, test-web waits only for build-web, not for build-api.

Try it: stage barrier versus needs

This stand-in computes when the pipeline finishes in two ways: with plain stages (each stage waits for the whole previous stage) and with needs (each job waits only for its dependencies). Durations are virtual seconds, and capacity is unlimited.

JS Compare stage barriers with a needs graph

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With stage barriers the pipeline needs 11 seconds (4, then 5, then 2); with needs it needs 9, because test-web begins at second 2, right after the short build-web. The longest chain of dependent jobs, here build-web, test-web and deploy, is the critical path; it sets the minimum possible time.

artifacts: files passed on and kept

An artifact is a set of files from a job that GitLab uploads when the job ends. Later jobs download them, and people can download them from the UI.

YAML
build-app:
  stage: build
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

unit-tests:
  stage: test
  script:
    - npm ci
    - npm test
  artifacts:
    when: always
    reports:
      junit: reports/junit.xml

Notes:

  • paths lists what to upload, relative to the project directory. expire_in stops artifacts from filling storage forever.
  • A later job gets the artifacts of earlier-stage jobs automatically (or of its needs); dependencies: narrows which jobs to fetch from.
  • when: always uploads even when the job failed, which you want for test reports and screenshots.
  • reports are special artifacts GitLab understands: junit shows test results in the merge request, coverage_report shows coverage, and there are others for security scanning.

cache: reuse downloaded dependencies

A cache keeps files such as node_modules or a package manager's download folder between pipeline runs, so you do not download the world every time.

YAML
default:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
    policy: pull-push

install-and-test:
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test

The key decides which cache a job uses; key: files: derives it from the hash of the listed files, so the cache changes exactly when the lock file does. policy: pull-push (the default) downloads at the start and uploads at the end; use policy: pull for jobs that only read. Cache and artifacts look alike but differ in purpose:

Artifacts Cache
Purpose Pass results to later jobs, keep outputs Speed up repeated downloads
Guaranteed Yes, within the pipeline No, best effort; a job must work without it
Scope One pipeline (and downloads) Across pipelines and branches by key

Never put build outputs in the cache to hand them to another job, and never depend on the cache being present.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A job has two rules. The first does not match; the second matches with when: never. What happens?

    The first matching rule decides, and when: never means the job is not added.
  2. 2. Why does a merge request pipeline need $CI_PIPELINE_SOURCE == "merge_request_event" in a job's rule?

    A rule that tests only CI_COMMIT_BRANCH never matches in a merge request pipeline, where that variable is empty.
  3. 3. What does needs: [build-web] on test-web change?

    needs builds a graph: a job starts when its named dependencies are done, without waiting for the rest of the previous stage.
  4. 4. Which is the right tool to pass the compiled dist/ folder from a build job to a later deploy job?

    Artifacts are guaranteed to arrive. A cache is best effort and meant for speeding up dependency downloads.
  5. 5. What does key: files: [package-lock.json] in a cache achieve?

    The key is derived from a hash of the listed files, so dependency changes create a new cache while unchanged lock files reuse the old one.

Summary

  • rules are evaluated top to bottom; the first match decides whether a job is added and with which when. Prefer rules to only/except.
  • workflow:rules controls whether the pipeline exists and avoids duplicate branch and merge request pipelines.
  • needs turns stages into a DAG so jobs start as soon as their dependencies finish; the critical path sets the lower bound for pipeline time.
  • Artifacts pass files between jobs and carry reports like JUnit; cache is a best-effort speed-up keyed by files such as the lock file.

Next: Includes, Templates and Environments.