Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 44 ยท Practice and Review

Common CI/CD Mistakes and Anti-Patterns

Learn the CI/CD anti-patterns that waste time and cause incidents: leaked secrets, unpinned actions, slow jobs, ignored flaky tests, manual steps and no rollback.

Most pipeline problems are not exotic. The same handful of mistakes appear in project after project: a secret printed in a log, an action pulled from a moving tag, a single 40-minute job that nobody dares touch, a flaky test that everyone re-runs until it turns green. Each one is easy to introduce, and each one costs more the longer it stays.

This lesson walks through the most common anti-patterns, explains why they hurt, and shows the better pattern next to each. The YAML is read-only, because real pipelines need a CI server. The runnable part is a stand-in: a small linter written in JavaScript that scans pipeline text for several of these smells. It is a teaching tool built on simple text patterns, not a replacement for real analysis tools. When you want to dig deeper into a topic, the relevant lessons are linked, for example Securing Your Pipeline and Making Pipelines Fast.

Why anti-patterns persist

Pipelines are code that few people own. They are written once under time pressure, copied between repositories, and only looked at when they break. A shortcut that saves ten minutes today (an unpinned version, a skipped check, a manual deploy) stays in place for years. The cure is to treat pipeline files like production code: review them, keep them small, and let a linter and automated updates watch for drift.

flowchart LR
  S["Shortcut taken<br/>under pressure"] --> C["Copied into other repos"]
  C --> F["Failure or incident"]
  F --> W["Workaround added<br/>(re-run, manual fix)"]
  W --> S

Anti-patterns feed themselves: each workaround makes the next shortcut look normal.

1. Secrets in logs and in code

The worst mistake is also the simplest: a token appears where others can read it. It happens in three ways. People echo a secret to debug a failing step. A script runs with shell tracing on (set -x), which prints every command including expanded secrets. Or a credential is committed to the repository because "it is a private repo".

YAML
# Anti-pattern
- run: |
    set -x
    curl -H "Authorization: Bearer ${{ secrets.API_TOKEN }}" https://api.example.com

CI systems mask known secret values in logs, but masking is a safety net, not a guarantee: a value that is transformed (base64, split, URL-encoded) is no longer recognized. The better pattern passes the secret through an environment variable, never prints it, and grants it only to the step that needs it:

YAML
- name: Call the API
  env:
    API_TOKEN: ${{ secrets.API_TOKEN }}
  run: curl -sS -H "Authorization: Bearer $API_TOKEN" https://api.example.com

If a secret ever reaches a log or a commit, treat it as compromised and rotate it. Deleting the log line is not enough. See Secrets, Variables and Environment Variables.

2. Unpinned actions, images and tools

uses: some-org/action@main runs whatever code is on that branch today, with your secrets in reach. Even a version tag such as @v3 can be moved by the owner, or by an attacker who gains access. The same applies to image: node:latest, which changes without warning and breaks builds on a day when nothing in your repository changed.

YAML
# Anti-pattern
- uses: some-org/deploy-action@main
- uses: some-org/lint-action@v2

For third-party actions, pin to a full commit SHA and note the version in a comment, then let a bot such as Dependabot or Renovate propose updates:

YAML
- uses: some-org/lint-action@3f2a9c1d5b7e4f60a81b2c3d4e5f60718293a4b5 # v2.4.1

For container images and tool versions, pin to an explicit version (or digest). Reproducible builds are boring in the best way. The trade-off is maintenance, which automation absorbs. See Pipeline Metrics and Maintenance.

3. One slow, monolithic job

A single job that installs, lints, runs every test, builds and packages is simple to write and painful to live with. It runs strictly one thing at a time, so feedback takes the sum of all parts. A failure at minute 38 means a 38-minute wait to learn something that a linter could have reported in 30 seconds. And one flaky step makes the whole job red.

The better pattern splits work along natural lines, orders cheap checks first, and runs independent jobs in parallel:

YAML
jobs:
  lint:
    runs-on: ubuntu-latest
    steps: [{ uses: actions/checkout@v4 }, { run: npm run lint }]
  unit:
    runs-on: ubuntu-latest
    steps: [{ uses: actions/checkout@v4 }, { run: npm run test:unit }]
  integration:
    needs: unit
    runs-on: ubuntu-latest
    steps: [{ uses: actions/checkout@v4 }, { run: npm run test:integration }]

Measure before you optimize: find the critical path (the longest chain of dependent work) and shorten that first. See Making Pipelines Fast.

4. Ignoring flaky tests

A flaky test passes and fails without code changes. The usual reaction is to add retries, or to teach the team to "just re-run it". Both erode trust. When red builds are routinely ignored, a real failure hides among the noise, and people start merging on red.

Better habits: quarantine the test (run it separately, do not block merges, and file an issue with an owner and a deadline), track the failure rate per test, and fix the cause. Common causes are shared state between tests, real time and random values, dependence on test order, and tests that depend on external services. Retries can be a temporary bandage for a known problem, but never a permanent policy; if you keep them, report every retry so the cost stays visible. See Test Stages in a Pipeline.

5. Manual steps in the middle

"Deploy is automatic, except that Dana runs a script on her laptop to apply the database migration." Manual steps are undocumented dependencies on a person. They cause delays, mistakes and a bus factor of one, and they make it impossible to reproduce a release.

Automate the step, or make it an explicit, recorded approval (an environment with required reviewers, a manual job with a clear name and a protected role). A manual gate is fine when a human decision is the point, such as approving a production release. A manual task is not. Document the remaining manual gates in the pipeline itself, where people will see them.

6. Long-lived branches and big-bang merges

Continuous integration means integrating continuously. A feature branch that lives for three weeks accumulates conflicts, drifts away from main, and arrives as one huge change that is hard to review and risky to ship. Its CI runs go green on a branch that is no longer representative of the product.

Prefer small pull requests merged within a day or two, and hide unfinished work behind feature flags so that incomplete code can reach main safely. Trunk-based development is the extreme form of this idea. Where long-lived release branches are needed, automate the back-merge so fixes are not lost.

7. No rollback plan

A pipeline that can only move forward turns every bad release into an emergency. If the only fix is "write a patch, run the whole pipeline again", recovery takes as long as a normal release, at the worst possible time.

Make rollback a first-class operation: keep previous artifacts (images tagged by commit SHA, packages for older versions), make the deploy job take a version as an input so that redeploying an old one is a normal run, and test the rollback path occasionally. Database changes need special care: use the expand/contract pattern so that old and new application versions can both run against the schema. See Deployment Strategies.

More smells worth knowing

Smell Why it hurts Better
command || true Hides real failures Handle the specific exit code
curl ... | bash in a step Runs unreviewed remote code Download, verify a checksum, then run
npm install in CI Can change the lockfile and the result npm ci
No timeout-minutes A hang burns minutes for hours A limit of a few times the normal run time
Broad token permissions A compromised step can do anything Declare minimal permissions
Deploy from any branch Unreviewed code reaches production Gate on branch and environment
Different build per environment Production runs untested bytes Build once, promote

Stand-in: an anti-pattern linter

The linter below checks pipeline text for several of these smells and prints a line per finding with its line number. It works on raw text with regular expressions, so it can be fooled (for example by a comment), and it knows nothing about YAML structure. Edit the sample or paste in your own YAML to see what it finds.

JS Pipeline anti-pattern linter

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The linter reports eight findings for the sample (six line-level and two file-level) and none for the clean snippet. Real tools in this space understand YAML and workflow semantics; this one only demonstrates how cheap it is to catch the most common smells automatically.

Stand-in: what a flaky test costs

Flakiness feels harmless until you do the arithmetic. If each test in a large suite fails randomly now and then, a clean run is far less likely than people expect. The second stand-in computes it.

JS The probability of a green run

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Even a tiny per-test flake rate adds up across a large suite. That is why quarantining and fixing flaky tests quickly is cheaper than living with retries.

A habit that prevents most of this

Once a quarter, read your pipeline files from top to bottom as if you were reviewing them for the first time. Ask five questions: Can a secret leak here? What runs that nobody pinned? What is the slowest path? Which failure do we ignore? How do we undo this deploy? Write down the answers, fix the top two, and schedule the rest.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is pinning a third-party action to a full commit SHA safer than using a tag?

    A tag is a movable label. A commit SHA identifies exact code, so an update must be a deliberate change you can review.
  2. 2. A test fails randomly once in every ten runs. What is the healthiest response?

    Retries and re-runs hide the problem and teach people to ignore red builds. Quarantine keeps merges unblocked while making the problem visible.
  3. 3. What is wrong with a deploy that depends on one person running a script on their laptop?

    Manual tasks create delays, mistakes and a single point of failure. Automate them, or turn real decisions into explicit approvals.
  4. 4. Which change makes rollback a normal operation?

    If previous versions are kept and addressable, redeploying one is just another run of the same pipeline.
  5. 5. What does adding || true after a test command do?

    The step exit code becomes zero regardless of the test result, so the pipeline cannot catch the failure.

Summary

  • Never print secrets or commit them; masking is a safety net, and a leaked secret must be rotated.
  • Pin actions, images and tool versions, and let a bot propose updates.
  • Split slow monolithic jobs, run independent work in parallel and put cheap checks first.
  • Treat flaky tests as bugs: quarantine, track and fix them instead of retrying forever.
  • Replace manual tasks with automation or explicit approvals, keep branches short-lived, and make rollback a normal run of the deploy job.
  • A text linter can catch many of these smells, but it only approximates what real analysis tools do.

Next: CI/CD Interview Questions