Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 44 ยท Performance and Operations

Pipeline Metrics and Maintenance

Measure delivery with the four DORA metrics, track flaky tests and runner cost, send pipeline notifications to chat and automate updates with Dependabot.

You cannot improve what you cannot see. A pipeline that "feels slow" or "seems flaky lately" is an opinion; a pipeline with numbers is a conversation you can have with your team. This lesson covers what to measure, how the best-known set of delivery metrics (the DORA metrics) are calculated, how to keep an eye on flaky tests and cost, how to send useful notifications, and how to keep the pipeline itself healthy over time by automating dependency and action updates.

Real metrics come from your CI system's API, your deployment records and your monitoring tools, none of which exist on this site, so the configuration is read-only. The runnable part is a Python stand-in that computes the four DORA metrics from a small list of deployment records. The arithmetic is the same whether you have 8 deployments or 8,000.

What to measure

A pipeline has two kinds of health. Pipeline health is about the machinery: how long runs take, how often they fail for reasons unrelated to the code, how much they cost. Delivery health is about outcomes: how quickly and safely your changes reach users. Measure both, and be careful about what you reward. A number that becomes a target tends to get gamed (counting tiny deployments to look fast, for example), so use metrics to start discussions, not to rank people.

The four DORA metrics

The DevOps Research and Assessment (DORA) research program found that four measurements separate high-performing delivery teams from low-performing ones. Two measure speed and two measure stability.

Metric Question it answers How to calculate it
Deployment frequency How often do we release to production? Count successful production deployments per day or week
Lead time for changes How long from commit to running in production? Median of (deploy time minus commit time)
Change failure rate What fraction of deployments cause a problem? Deployments needing a fix, rollback or hotfix, divided by all deployments
Time to restore service How long to recover from a failed deployment? Median of (restored time minus failure time)
flowchart LR
    C["Commit"] -->|"lead time"| D["Deploy to production"]
    D -->|"frequency: how often"| D
    D --> F{"Caused a failure?"}
    F -- "yes: counts toward failure rate" --> R["Restore service"]
    R -->|"time to restore"| OK["Healthy again"]
    F -- no --> OK

Lead time and frequency describe speed; failure rate and time to restore describe stability.

Some details that trip people up:

  • Medians beat averages. One change that sat in review for three weeks distorts an average. Use the median, and look at the spread too.
  • Define "failure" in advance. Typical definitions: a rollback, a hotfix deployed within some window, or an incident linked to the deployment. Write the definition down so everyone counts the same way.
  • Lead time start. The most common start is the first commit of the change (or merge into main, for a narrower "pipeline lead time"). Pick one and stay consistent.
  • Speed and stability are not a trade-off. The research shows that good teams are good at both. Small, frequent releases are easier to test and easier to roll back.

Stand-in: DORA from deployment records

In practice you export deployment records: when the change was committed, when it reached production, whether it failed, and when service was restored. This stand-in computes all four metrics with Python's datetime and statistics modules. The records are fixed sample data, so the output is deterministic.

Python Compute the four DORA metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the slowest lead time is much larger than the median. The median tells you the typical experience; the slowest change tells you where to investigate. If you lean toward a single number, pick the median. If the slowest changes are common, look for review queues, manual approvals or long-lived branches (see Common CI/CD Mistakes and Anti-Patterns).

Pipeline health metrics

DORA tells you about delivery. These metrics tell you about the pipeline itself:

  • Duration (p50 and p90). The median and the 90th percentile of wall-clock run time. Rising p90 is an early warning, covered in Making Pipelines Fast.
  • Queue time. How long jobs wait for a runner. Large values mean too few runners.
  • Success rate on the main branch. Main should be green almost always. A main branch that is red for hours blocks everybody.
  • Mean time to green. How long a broken main stays broken.
  • Flaky rate. The share of failures that pass on re-run with no code change.
  • Cost. Billed runner minutes or infrastructure spend per week, and per repository or team.

Flaky tests

A flaky test passes and fails on the same code. Flakiness is poison: people learn to ignore red builds and re-run until green, which hides real failures. To track it, record every test result along with the commit. A test that both passed and failed for the same commit is flaky by definition. List the worst offenders and fix or quarantine them: move them to a non-blocking job, link a ticket to each, and put a deadline on the quarantine. Test runners and reporting tools can produce JUnit XML, which gives you per-test data for this analysis. An automatic "retry on failure" setting is a temporary painkiller at best; retried tests need to be counted and reported, not hidden.

Runner cost

GitHub bills hosted runner minutes with a multiplier by operating system (Linux is the cheapest; macOS and Windows cost several times as much), and larger runners cost more per minute. Minutes also add up through matrices: a 10-minute job in a 3-by-3 matrix is 90 minutes of billed time even though you wait only 10. Ways to control cost:

  • Cancel superseded runs with concurrency.
  • Use path filters and affected-project detection.
  • Run expensive matrix entries (macOS, Windows, old versions) only on main or on a schedule.
  • Set timeout-minutes so a hung job cannot burn hours.
  • Consider self-hosted runners when volume is high (see Self-Hosted Runners and Runner Costs); remember they cost maintenance time too.

Notifications that people read

A notification is only useful if someone acts on it. Too many alerts teach people to mute the channel. Good rules:

  • Notify on main branch failures and on deployments, not on every pull request run (the author already sees those).
  • Notify on state changes: first failure and recovery, not every failing run in a row.
  • Include the essentials: repository, branch, commit, author, a one-line cause if known, and a link to the run.
  • Send to the people who can act, a team channel for main, the author for their own pull request.

Chat tools usually provide an incoming-webhook URL, which you store as a secret. Here is a minimal workflow step that posts to such a webhook when the job fails:

YAML
notify-failure:
  needs: [build, test]
  if: failure() && github.ref == 'refs/heads/main'
  runs-on: ubuntu-latest
  steps:
    - name: Post to chat
      env:
        WEBHOOK_URL: ${{ secrets.CHAT_WEBHOOK_URL }}
        RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
        ACTOR: ${{ github.actor }}
      run: |
        curl --fail --silent --show-error \
          -H "Content-Type: application/json" \
          -d "{\"text\": \"CI failed on main (triggered by ${ACTOR}): ${RUN_URL}\"}" \
          "$WEBHOOK_URL"

Two details: the values reach the script through env rather than being pasted into the shell text, which is the safe habit explained in Securing Your Pipeline; and if: failure() makes the job run only when a needed job failed, as described in Expressions and Contexts. The exact JSON shape depends on your chat tool, so check its documentation. Jenkins has the post { failure { ... } } section, and GitLab has built-in integrations for chat services.

Keeping the pipeline up to date

Pipelines rot. Actions release security fixes, base images get patched, Node versions reach end of life, and a workflow nobody touched for two years quietly depends on deprecated things. The fix is to automate the boring part. Two popular tools open pull requests that bump versions for you, and CI then tests each bump like any other change.

Dependabot is built into GitHub. A configuration file tells it which ecosystems to watch:

YAML
# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
    groups:
      actions:
        patterns: ["*"]

  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 5
    groups:
      minor-and-patch:
        update-types: ["minor", "patch"]

The github-actions ecosystem updates the uses: lines in your workflows, and it can update SHA-pinned actions too, which is why pinning and automation go well together. groups bundle several updates into one pull request to reduce noise.

Renovate is a similar, more configurable tool that works on GitHub, GitLab and other platforms and can be self-hosted. It reads a renovate.json file and supports rules such as automerging minor updates when tests pass. Which one you choose matters less than turning one of them on.

Pair automation with a safety net: required status checks, a good test suite and a rule that nothing merges red. Automated update pull requests are only as safe as the checks that gate them. A short maintenance routine helps too: once a quarter, review runner images, language versions, secrets that should be rotated and workflows that no longer have an owner.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which pair of DORA metrics measures delivery speed?

    Deployment frequency and lead time for changes are the speed metrics. Change failure rate and time to restore are the stability metrics.
  2. 2. Why is the median usually preferred over the average for lead time?

    Lead times are skewed by outliers such as a change that waited weeks. The median is robust against them.
  3. 3. Eight deployments, of which two needed a rollback or hotfix. What is the change failure rate?

    Two failures divided by eight deployments is 0.25, or 25 percent.
  4. 4. How do you recognize a flaky test from your test records?

    Same code, different results is the definition of flakiness. Tracking results per commit reveals it.
  5. 5. What does the github-actions ecosystem in dependabot.yml update?

    It watches workflow files and opens pull requests that bump the actions they use.

Summary

  • Measure both pipeline health (duration, queue time, main-branch success, flakiness, cost) and delivery health.
  • The four DORA metrics are deployment frequency, lead time for changes, change failure rate and time to restore service; use medians and define "failure" clearly.
  • Track flaky tests per commit, quarantine them with a deadline, and never hide retries.
  • Control cost with cancellation, path filters, selective matrices and timeouts.
  • Send notifications on main failures, deployments and state changes, with links and context, and pass values through env.
  • Automate updates of actions and dependencies with Dependabot or Renovate, gated by good checks.

Now put the pieces together in your first full project: Project: CI for a Node.js Library.