Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท CI, Pipelines and Review Apps

Heroku CI: Test Runs

Run your test suite on Heroku CI: enable it for a pipeline, configure the test environment in app.json, add Postgres and Redis for tests, and read results from the CLI.

Heroku CI is a hosted continuous integration service built into pipelines. When you push a commit or open a pull request, Heroku creates a temporary test app, installs your dependencies with the same buildpacks as production, runs your test command inside a one-off test dyno, and reports a status to GitHub. The advantage over an external CI is that tests run in the same environment as production: same stack, same buildpack, same add-on types.

This lesson shows how it is configured. Heroku CI cannot run on this site, so configuration files and commands are read-only; the Python stand-ins let you practice reading and checking app.json.

Turning it on

Prerequisites: an app in a pipeline, a connected GitHub repo, and CI enabled for the pipeline (in the dashboard, the Tests tab of the pipeline; or heroku ci:... commands). Heroku CI is a paid feature, billed by the time test dynos run, so check the current pricing and set limits before enabling it for a big team.

Configuring tests in app.json

Tests are described in the app.json file in your repository's root. The environments.test section sets the test script, environment variables, add-ons and buildpacks:

JSON
{
  "environments": {
    "test": {
      "scripts": {
        "test": "pytest -q"
      },
      "env": {
        "APP_ENV": "test"
      },
      "addons": [
        "heroku-postgresql:in-dyno",
        "heroku-redis:in-dyno"
      ]
    }
  }
}
  • scripts.test is the command run on the test dyno; its exit code decides pass or fail (0 passes).
  • env sets variables for the test run, so no production secrets are needed.
  • addons with the in-dyno plan starts a Postgres or Redis inside the test dyno, free of extra add-on charges and fast to create. DATABASE_URL and REDIS_URL point to them.
  • Optional buildpacks and a setup script (for example to load fixtures) are also available. A scripts.test-setup script runs before the tests, and scripts.postdeploy belongs to review apps.

Test dynos use their own size (performance-m by default, set with formation), and a run is limited in time, so keep suites fast.

A typical flow

flowchart LR
    P["git push / PR"] --> B["Build test slug"]
    B --> S["Start test dyno<br/>with in-dyno Postgres"]
    S --> T["Run scripts.test"]
    T -->|exit 0| OK["Status: success"]
    T -->|exit non-zero| FAIL["Status: failure"]
    OK --> G["GitHub check passes"]

Heroku builds, runs the test script, and reports the exit code as a GitHub check.

CLI

Terminal
heroku ci:run --pipeline my-pipeline
heroku ci:last --pipeline my-pipeline
heroku ci:info 1 --pipeline my-pipeline
heroku ci:config --pipeline my-pipeline
heroku ci:config:set DEBUG=0 --pipeline my-pipeline
heroku ci:debug --pipeline my-pipeline

ci:run runs the tests from your local working copy (handy before pushing). ci:last shows the latest run, ci:info a specific run by number. ci:config and ci:config:set manage variables used only by CI. ci:debug opens a shell in a test environment so you can reproduce a failure.

Writing a good test script

A test script is just a shell command. A few habits make CI trustworthy:

  1. Fail fast and loudly. Use a non-zero exit code on any failure (most test runners do this).
  2. Be deterministic. Seed random generators, freeze clocks, avoid network calls to real services; Heroku test dynos have outbound network access, but depending on it makes tests flaky.
  3. Run migrations first. A script like python manage.py migrate && pytest makes the in-dyno database ready.
  4. Keep the suite under a few minutes; slow CI makes people skip it. Split slow end-to-end tests into a separate job on a schedule.
  5. Print enough on failure to debug without logging into the dyno.
Python Stand-in: check an app.json test section

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The checker accepts the first document and flags the second because a paid Standard database does not belong in a test environment.

What CI proves and what it does not

CI proves that the code passes your tests in a production-like environment. It does not prove performance under load, correctness of real third-party integrations, or that your tests are any good. Combine it with code review and a staging smoke test.

Python Stand-in: exit code decides the status

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where do you configure the test script for Heroku CI?

    app.json describes the test environment: script, env vars, add-ons and buildpacks.
  2. 2. What does the in-dyno plan for an add-on mean?

    in-dyno Postgres or Redis starts inside the temporary test dyno and disappears after the run.
  3. 3. What decides whether a Heroku CI run passes?

    Exit code 0 is success; anything else is failure.
  4. 4. Which command runs tests from your local working copy on Heroku CI?

    heroku ci:run uploads your working tree and runs the test script in the CI environment.

Summary

  • Heroku CI runs tests in a temporary app on a test dyno, with the same buildpacks as production.
  • app.json environments.test sets script, env and in-dyno add-ons; the exit code decides the result.
  • ci:run, ci:last, ci:config and ci:debug manage runs.
  • Keep tests fast and deterministic; pair CI with review and a staging check.

Next: Review Apps and app.json