Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 44 ยท Performance and Operations

Debugging Failing Pipelines

Find out why a CI pipeline fails: read logs from the first error, turn on debug logging, reproduce runs locally and fix the most common failure causes.

Every pipeline fails sooner or later. A test breaks, a secret expires, a runner runs out of memory, a YAML file has one space too many. The skill that separates a calm engineer from a frustrated one is not knowing all the answers, it is having a method: read the evidence in the right order, form a guess, test it cheaply, and only then change something.

This lesson gives you that method. You will learn how to read a log, how to turn on extra detail, how to reproduce a failure on your own machine, and what the most common causes look like. The commands are read-only because they target tools that are not available here, but the stand-in is something you can run and extend: a small log scanner that classifies failure causes, which is also a good way to understand why those causes leave recognizable fingerprints.

A debugging method

  1. Find the first real error. Logs are read top to bottom, but failures cascade. The step that went red is often reporting a consequence. The line that caused it may be earlier. Search for the first error, fatal, failed or non-zero exit code.
  2. Say what you expected. "The test command should have found 200 tests" or "the file should be at dist/app.js." Debugging is comparing expectations with reality.
  3. Change one thing at a time. Several simultaneous changes teach you nothing, even when they work.
  4. Reproduce when you can. A failure you can trigger locally takes seconds per attempt instead of minutes.
  5. Fix the cause, then the process. If the failure was confusing, improve the message or the check so the next person finds it faster.
flowchart TD
    A["Pipeline is red"] --> B["Find the first error in the log"]
    B --> C{"Does the message explain it?"}
    C -- yes --> D["Fix and re-run"]
    C -- no --> E["Turn on debug logging"]
    E --> F{"Reproducible locally?"}
    F -- yes --> G["Iterate locally, then push"]
    F -- no --> H["Compare runner and local environments"]
    H --> D

Start with the log; add detail only when the message is not enough.

Reading logs well

Logs are long, and CI systems help you navigate them. In GitHub Actions each step is a collapsible group, a red cross marks the failed step, and there is a search box for the log. In GitLab the job log shows collapsible sections. In Jenkins the console output shows everything in sequence, and Blue Ocean or the pipeline stage view marks the stage that failed.

Things to look for:

  • The exit code. Remember from Anatomy of a Pipeline that a non-zero exit code is what fails a step. Some codes carry meaning: 127 is "command not found", 126 is "found but not executable", 137 usually means the process was killed (often out of memory), and 143 means it received a termination signal, which is common after a timeout or cancellation.
  • The command that actually ran. Check the expanded command (after variables and expressions were filled in). A typo in an environment variable name shows up here.
  • Warnings before the failure. Deprecation notices and "using cache key X" lines often explain surprises.
  • Timing. A step that normally takes 10 seconds and took 6 minutes tells you where to look.

Turning up the detail

GitHub Actions

When you re-run a job from the web page, there is a checkbox to enable debug logging. You can also make it permanent for a repository by setting a secret or variable named ACTIONS_STEP_DEBUG to true, which prints extra diagnostic lines from steps, and ACTIONS_RUNNER_DEBUG to true for runner diagnostics. Debug lines are printed with a ::debug:: prefix in the raw log; actions can emit their own with the echo "::debug::message" workflow command.

You can also add a throwaway diagnostic step. Be careful never to print secrets (see Secrets, Variables and Environment Variables):

YAML
- name: Show the environment
  run: |
    pwd
    ls -la
    echo "node: $(node --version)"
    echo "event: ${{ github.event_name }}  ref: ${{ github.ref }}"

To see the full objects behind the contexts, print them as JSON:

YAML
- name: Dump the github context
  env:
    GITHUB_CONTEXT: ${{ toJSON(github) }}
  run: echo "$GITHUB_CONTEXT"

Passing the value through env instead of pasting ${{ }} directly into the script avoids a class of injection problems explained in Securing Your Pipeline. Remove such steps when you are done.

GitLab CI/CD

Setting the variable CI_DEBUG_TRACE: "true" makes the runner print every command and variable expansion. That output can include the values of secret variables, so use it only in a private project and delete the logs afterwards. Safer first steps are adding set -x to a script block to trace shell commands, or printing the file listing and tool versions.

YAML
debug-job:
  script:
    - set -x
    - ls -la
    - node --version

Jenkins

Jenkins has Replay, which lets you edit the Pipeline script of a past run and run it again without committing, which is a quick way to test a fix. Add set -x to sh steps for shell tracing, and use the echo step for values. The console output is the single source of truth, and the Pipeline Steps view shows what each step did.

Reproducing the failure locally

The ideal fix cycle happens on your machine. There are three levels of fidelity:

Run the same commands. Most failures are in your build or test commands, not in the CI system. Copy the failing run lines and execute them from a fresh checkout in a clean directory. Use the same versions of the language and tools that the pipeline installs. Many "works on my machine" failures disappear once you use npm ci rather than npm install, because CI installs exactly what the lock file says.

Run in the same image. If the job runs in a container (or the runner image is Linux and your laptop is not), start that container and run the commands inside it:

Terminal
docker run --rm -it -v "$PWD":/work -w /work node:22-bookworm bash
# inside the container:
npm ci
npm test

Run the workflow itself. The community tool act runs GitHub Actions workflows locally in Docker containers:

Terminal
act -l                    # list the jobs it finds
act pull_request -j test  # run the job named "test" as a pull_request event

act is a very useful approximation but not a perfect copy: its default images are smaller than GitHub's runner images, some contexts and services differ, and features such as OIDC, caches and hosted artifacts may not behave the same. Treat a pass in act as a good sign and a pass on GitHub as the proof.

For GitLab, older versions of the runner had a gitlab-runner exec command for running one job locally; it was removed in newer major versions. A dependable replacement is to start the job's image: with Docker and run its script lines by hand, as in the example above. For Jenkins, run the shell commands of the failing stage in the same Docker image the agent uses.

The common failures

Most pipeline failures fall into a short list. Learn their fingerprints and you will recognize them in seconds.

Symptom in the log Likely cause First thing to try
Permission denied on a script File lacks the executable bit, or the token cannot write chmod +x and commit it with git update-index --chmod=+x; check permissions:
Resource not accessible by integration The GITHUB_TOKEN has too few permissions Add the specific permission the job needs
command not found (exit 127) Tool not installed or not on PATH on the runner Add the setup step; print which tool
No such file or directory Wrong working directory, or a case-sensitive path on Linux Print pwd and ls; check the exact case
Error: Cannot find module / dependency missing Stale or wrong cache, or lock file out of date Bust the cache key; use npm ci
Killed, exit 137 Out of memory Reduce parallelism, raise limits, use a larger runner
The job running on runner ... has exceeded the maximum execution time Hang or timeout Add timeout-minutes, find what waits for input
Invalid workflow file / mapping values are not allowed here YAML indentation or quoting mistake Validate YAML; see YAML for Pipeline Authors
Works on main, fails on a fork PR Secrets are not given to forks Expected; redesign the check
Passes locally, fails in CI Different versions, case, time zone, missing env var, test order Match versions; run the exact CI commands
Fails once, passes on re-run Flaky test or network blip Track it; do not just re-run forever

Two of these deserve a closer look:

  • Cache problems. A restored cache can contain outdated files that make a build behave differently from a clean one. If a failure looks impossible, rerun once with the cache disabled or change the key. If that fixes it, you have found the culprit.
  • Timeouts. A default job limit of several hours means a hung step can burn a lot of minutes before anyone notices. Always set timeout-minutes on jobs, and consider timeouts on individual steps.

Stand-in: a log scanner

Humans scan logs by pattern: they know that "ENOSPC" means a full disk. A script can do the same. This stand-in reads a sample log, finds the first matching failure fingerprint, and suggests where to look. It is the same idea that tools use to highlight errors and group similar failures. The rules here are only a handful; real systems keep many more, and a match is a hint, not a verdict.

JS Classify the cause of a failing log

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice two design points. First, the scanner reports the first matching line, which follows the "find the first real error" rule. Second, the last log shows the honest limit of automation: an unknown failure still needs a human, and a new rule should be added afterwards so the next occurrence is recognized.

Stand-in: exit codes in plain words

Exit codes are the language of CI, and a few of them are worth knowing by heart. This second stand-in turns a code into a sentence and shows the common shell convention that "128 plus N" means "killed by signal N".

JS What does this exit code mean?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

After the fix

Fixing the red build is half the job. Ask three more questions: Could the message have told us faster? Could a check have caught this before merge? Is this a repeat? If a failure appears again and again, write it down, add a rule to your tooling, and consider whether the pipeline design (not the person) needs to change. The next lesson looks at measuring exactly this kind of health: Pipeline Metrics and Maintenance.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A step fails with exit code 137. What is the most likely explanation?

    137 is 128 plus signal 9 (SIGKILL). On CI runners this usually means the out-of-memory killer ended the process.
  2. 2. Where should you look first in a long failing log?

    Errors cascade. The final line usually just says the step exited non-zero, while the cause is earlier.
  3. 3. Why is CI_DEBUG_TRACE risky in GitLab?

    The trace prints expanded commands and variables, which may include secrets. Use it only in private projects and remove the logs.
  4. 4. A pull request from a fork fails at a step that needs a secret, while the same workflow passes on main. What is the likely reason?

    GitHub withholds secrets from fork pull requests for safety, so the step sees an empty value.
  5. 5. What is the main limitation of running a workflow locally with act?

    act is a handy approximation. A local pass is a good sign, but the real runner is the final test.

Summary

  • Use a method: find the first real error, state your expectation, change one thing at a time, reproduce, then fix the cause.
  • Exit codes carry meaning: 126 not executable, 127 not found, 137 killed (often memory), 143 terminated.
  • Turn up detail carefully: re-run with debug logging in GitHub, set -x or CI_DEBUG_TRACE (private projects only) in GitLab, Replay in Jenkins.
  • Reproduce locally by running the same commands, running inside the job's Docker image, or using act for GitHub workflows, knowing it is an approximation.
  • Most failures are permissions, paths, missing tools, stale caches, memory, timeouts or YAML mistakes; always set timeout-minutes.
  • After fixing, improve the message or add a check so the failure is faster to find next time.

Next: Pipeline Metrics and Maintenance.