Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 41 ยท Packages and Deployment

Deploying dbt: Jobs and Scheduling

Run dbt in production: choose job commands, schedule with cron, dbt Cloud, Airflow or GitHub Actions, use source freshness, retries and run artifacts.

Deploying dbt means running it on a schedule, in an environment nobody is typing into, and making sure someone notices when it breaks. There is nothing to "deploy" in the web-server sense: dbt is a command-line tool that connects to the warehouse, runs SQL and exits. A production deployment is therefore a job: a set of dbt commands, a trigger, and alerts.

What a production job runs

The simplest job is a single command:

Terminal
dbt build --target prod

dbt build runs seeds, models, snapshots and tests in dependency order, and skips anything downstream of a failure. A fuller production job often looks like this:

Terminal
dbt deps
dbt source freshness --target prod
dbt build --target prod --exclude tag:weekly
dbt docs generate --target prod
  1. dbt deps installs packages.
  2. dbt source freshness checks that source data is recent (see below).
  3. dbt build does the work.
  4. dbt docs generate refreshes documentation to publish.

You can run different jobs for different needs: an hourly job for tag:hourly models, a nightly full build, a weekly --full-refresh of incremental models.

Source freshness

If the upstream pipeline stalls, dbt would happily rebuild from old data and nobody would know. Declare how fresh each source must be:

YAML
sources:
  - name: shop
    database: raw
    loaded_at_field: _loaded_at
    freshness:
      warn_after: {count: 12, period: hour}
      error_after: {count: 24, period: hour}
    tables:
      - name: orders
      - name: customers
        freshness: null   # do not check this one

dbt source freshness runs select max(_loaded_at) per source table and compares it with now. The check, written out in SQL:

SQL Freshness check in SQL

Press Run (or Ctrl + Enter) to see the result here.

The newest row is about 16 hours old, which exceeds the 12-hour warn threshold but not the 24-hour error limit, so the result is WARN. Running freshness before the build means your marts are not silently stale.

Ways to schedule

Option Good for
dbt Cloud jobs Managed scheduling, run history, alerts, hosted docs, CI on pull requests
Cron / systemd timer A single server, simplest possible start
GitHub Actions / GitLab CI schedule Teams already using Git CI; simple and cheap
Airflow, Dagster, Prefect When dbt is one step in a larger pipeline (ingest, then dbt, then reverse ETL)
Warehouse-native schedulers Some warehouses can run dbt projects as scheduled tasks

A scheduled GitHub Actions workflow is only a few lines:

YAML
name: nightly-dbt
on:
  schedule:
    - cron: "0 3 * * *"
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install dbt-postgres
      - run: dbt deps && dbt build --target prod
        env:
          DBT_PROD_USER: ${{ secrets.DBT_PROD_USER }}
          DBT_PROD_PASSWORD: ${{ secrets.DBT_PROD_PASSWORD }}

With an orchestrator, trigger dbt right after ingestion finishes instead of guessing a clock time, and let the orchestrator handle retries and dependencies.

flowchart LR
  I["Ingest raw data"] --> F["dbt source freshness"]
  F --> B["dbt build"]
  B --> D["dbt docs generate"]
  B -->|"failure"| A["Alert: Slack / email"]
  D --> H["Publish docs site"]

A typical nightly production job.

Run artifacts

Every command writes JSON files to target/:

  • manifest.json: the whole project graph.
  • run_results.json: status, timing and rows of every node in the last run.
  • sources.json: freshness results.
  • catalog.json: from docs generate.

Store them: upload to object storage after each production run. They power slim CI (see CI with state selection), monitoring dashboards and alerts such as "a model is 3x slower than usual".

Retries and reruns

If one model fails, you do not have to rerun everything:

Terminal
dbt retry

dbt retry resumes from the failed nodes using the previous run_results.json. Combined with dbt build, this saves a lot of time on large projects. Also size threads to what the warehouse can handle, and avoid peak hours for heavy jobs.

Production checklist

  • A dedicated production user with write access only to production schemas.
  • Alerts on job failure (and on warnings if they matter).
  • Pinned package versions and a pinned dbt version.
  • Jobs are idempotent: running them twice produces the same result.
  • A documented rollback plan: Git revert, then rerun the affected models with --full-refresh when needed.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command runs seeds, models, snapshots and tests in dependency order?

    dbt build combines the other commands and skips nodes downstream of a failure.
  2. 2. What does dbt source freshness check?

    It compares max(loaded_at_field) with the current time against warn_after and error_after.
  3. 3. What does dbt retry do?

    It uses run_results.json to resume from the failure point.
  4. 4. Why upload target/ artifacts after production runs?

    manifest.json and run_results.json describe the production state, which other tools and jobs consume.

Summary

  • A deployment is a scheduled job that runs dbt build, ideally after a freshness check.
  • Schedule with dbt Cloud, cron, CI schedules or an orchestrator.
  • Store artifacts; use dbt retry to resume after failures.
  • Alert on failures and keep production credentials separate.

Next: CI with Slim CI and State Selection