Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท CI and Best Practices

JMeter in CI/CD: GitHub Actions, Jenkins, Docker and Maven

Add JMeter performance tests to a delivery pipeline with GitHub Actions, Jenkins, GitLab CI, Docker and the Maven plugin, and detect regressions by comparing with a baseline.

Performance regressions are rarely introduced on purpose. A new query, an extra network call, a library upgrade: each looks harmless in code review. The only way to catch them early is to measure on every change or every night. This lesson shows how to wire JMeter into a pipeline and how to decide automatically whether a build is slower than before. JMeter cannot run on this site, so the pipeline files are shown as read-only text, and the runnable code is a Python stand-in for the baseline comparison.

What belongs in a pipeline

Not every test fits every build. A common split:

Stage Test Duration When
Smoke 1 to 5 users, one iteration of the key flows under 2 minutes every commit or pull request
Load (short) realistic load, a few minutes 5 to 15 minutes every merge to main, or nightly
Full load / stress / soak peak load, ramp to failure, hours at steady load 30 minutes to many hours scheduled (nightly, weekly, before release)

Pipelines are for regression detection, not for finding the maximum capacity. Keep the environment, data and load shape the same each time so the numbers are comparable.

GitHub Actions

YAML
name: perf-smoke
on:
  pull_request:
  schedule:
    - cron: "0 3 * * *"
jobs:
  jmeter:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run JMeter in Docker
        run: |
          docker run --rm -v "$PWD:/work" -w /work justb4/jmeter:5.6.3 \
            -n -t plans/api.jmx -Jhost=${{ vars.STAGING_HOST }} -Jthreads=20 -Jduration=180 \
            -l out/results.jtl -j out/jmeter.log -e -o out/report
      - name: Check thresholds
        run: python scripts/check_jtl.py out/results.jtl
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: jmeter-report
          path: out/

Notes: the image tag pins a version, if: always() uploads the report even when the gate fails, and the staging host is a repository variable. (Image names change over time: check that the image you choose is maintained, or build your own from the Apache binary distribution.)

Jenkins

With a declarative pipeline:

GROOVY
pipeline {
  agent { label 'perf' }
  stages {
    stage('Smoke') {
      steps {
        sh 'rm -rf out && mkdir out'
        sh 'jmeter -n -t plans/api.jmx -Jthreads=20 -Jduration=180 -l out/results.jtl -e -o out/report'
        sh 'python3 scripts/check_jtl.py out/results.jtl'
      }
      post {
        always { archiveArtifacts artifacts: 'out/**', fingerprint: true
                 publishHTML(target: [reportDir: 'out/report', reportFiles: 'index.html', reportName: 'JMeter Dashboard']) }
      }
    }
  }
}

The Performance plugin can also parse JTL files and mark a build unstable or failed using thresholds and show trend graphs. The HTML Publisher plugin shows the dashboard report inside the build page.

GitLab CI

YAML
perf:
  stage: test
  image: justb4/jmeter:5.6.3
  script:
    - jmeter -n -t plans/api.jmx -l out/results.jtl -e -o out/report
  artifacts:
    when: always
    paths: [out/]

Maven and Gradle

For Java teams, the jmeter-maven-plugin runs tests during the verify phase. Test plans sit in src/test/jmeter, and the plugin downloads JMeter itself. This keeps the JMeter version in the pom.xml, which makes builds reproducible:

XML
<plugin>
  <groupId>com.lazerycode.jmeter</groupId>
  <artifactId>jmeter-maven-plugin</artifactId>
  <version>3.8.0</version>
  <executions>
    <execution><id>jmeter-tests</id><goals><goal>jmeter</goal></goals></execution>
    <execution><id>jmeter-check-results</id><goals><goal>results</goal></goals></execution>
  </executions>
</plugin>

The results goal fails the build when requests fail. Check the plugin's current version before copying.

Comparing with a baseline

A single run says little: "p95 = 830 ms" might be great or terrible. What matters is the change against a baseline from a previous successful run (the last release, or a rolling median of recent runs). Store the summary (a small JSON file) as an artifact or in a database, and compare after each run. Allow some tolerance because load tests are noisy (5 to 10 percent is common).

Python Stand-in: regression check against a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Test data and environments in pipelines

  • Environment: use a stable staging environment that resembles production in size and configuration; testing on a shared, constantly changing environment produces noise.
  • Data: reset or re-seed the database before each run so each run starts from the same state, and generate enough unique data for the planned load.
  • Secrets: pass credentials with CI secrets and properties (-Jpassword=$PERF_PASSWORD), never inside the .jmx or the repository.
  • Isolation: run the load generator in a different machine or container than the application under test.
  • Notifications: post a summary (p95, errors, link to the report) to chat when the gate fails.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which kind of test suits every pull request?

    Pull-request stages must be fast, so keep them to smoke or short checks and run heavier tests on a schedule.
  2. 2. Why upload the report with if: always()?

    A failed gate is when you most need the report to see what went wrong.
  3. 3. What is the benefit of comparing with a baseline?

    A change relative to a known good run signals regressions even when absolute numbers look acceptable.
  4. 4. Where should passwords for the test user come from?

    Keep secrets out of the repository and plans; inject them at run time.

Summary

  • Use short smoke tests on commits and longer load tests on schedules.
  • Run JMeter in non-GUI mode in Docker, Jenkins, GitHub Actions or GitLab CI, archive the JTL and the report.
  • Add a threshold gate and compare with a baseline with a tolerance for noise.
  • Control the environment, data, secrets and JMeter version.

Next: Test Plan Best Practices