Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 41 Β· Getting Started

How dbt Works: Architecture, Core, Cloud and Fusion

Understand what happens inside a dbt command (parse, compile, execute), how the dependency graph is built, and how dbt Core, dbt Cloud and Fusion differ.

When you type dbt run, a lot happens before the first table is created. Knowing the stages helps you debug errors, because every dbt error belongs to one of them: a parse problem (a broken YAML or Jinja file), a compile problem (a bad ref), or an execution problem (the warehouse rejected your SQL). This lesson walks through the pipeline and then compares the three ways to run dbt.

The life of a dbt command

From command to warehouse

  1. Parse
Read every file in the project
  1. Build the DAG
Link models through ref and source
  1. Compile
Render Jinja into plain SQL
  1. Execute
Send SQL to the warehouse

Parse and compile need no warehouse connection for most steps; execute does.

1 / 2

Parse. dbt reads dbt_project.yml, every .sql and .yml file under the configured paths, and macros. Jinja is partly evaluated here, just enough to learn configs, ref() calls and source() calls. Syntax errors in YAML or unbalanced Jinja tags fail here, before any connection.

Build the graph. Each model, seed, snapshot, source and test becomes a node. Every ref('x') becomes an edge from x to the model using it. The result is a DAG: directed (data flows one way) and acyclic (no circular dependencies).

flowchart LR
  A["source: customers"] --> B["stg_customers"]
  C["source: orders"] --> D["stg_orders"]
  B --> E["int_customer_orders"]
  D --> E
  E --> F["dim_customers"]
  D --> G["fct_orders"]
  F -.-> H["test: unique customer_id"]

A small DAG: sources feed staging models, which feed an intermediate model and marts. Tests hang off the nodes they check.

Compile. For each model dbt renders the Jinja into pure SQL. It knows the target, variables and every macro, so by now ref('stg_orders') is "analytics"."stg_orders".

Execute. dbt walks the DAG with a pool of threads (the threads setting). A model starts when all its parents have finished. It sends the CREATE ... AS statement to the warehouse, records success or failure, and moves on.

The DAG is the heart of dbt

You never state the order of models. You only declare dependencies by using ref() and source(), and dbt derives the order. That has real benefits:

  • Adding a new model in the middle of a chain needs no change to a scheduler.
  • Independent branches run in parallel automatically.
  • The lineage graph in the generated docs comes for free.
  • Selecting +fct_orders means "this model and everything it needs", which is just a graph query.

Circular references are rejected at the graph stage, with a message naming the cycle.

Stand-in: ordering a graph yourself

Here is the algorithm dbt uses to turn dependencies into a run plan, written in plain Python (not dbt). It groups models into waves that could run in parallel, and detects cycles.

Python Stand-in: waves of parallel work

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With threads: 4, wave 1 builds all three staging models at the same time. The cycle at the end is the situation that produces dbt's "Found a cycle" error.

Fusion's new angle

The stages above are the same in every dbt engine, but dbt Fusion does some of them differently: it understands SQL itself instead of only rendering text. That lets it catch many mistakes (an unknown column, a type mismatch) at compile time, before running anything in the warehouse.

dbt Core, dbt Cloud and Fusion compared

dbt Core dbt Cloud (dbt platform) dbt Fusion
What it is Open-source Python CLI Hosted product around dbt New engine, written in Rust
Cost Free Paid plans (with a free developer tier) Free CLI for local use, with platform features
You run it On your laptop, a server, CI In the browser or CLI with a managed service Local CLI, VS Code extension, or in the platform
Scheduling You bring your own (cron, Airflow, GitHub Actions) Built-in job scheduler Same options as Core or Cloud
IDE Any editor Web IDE, Studio VS Code extension with a language server
Extras Packages, docs CI on pull requests, alerts, semantic layer, catalog Faster parsing, static analysis

All three read the same project: dbt_project.yml, models, YAML, macros. The rest of this course uses commands and files that work in Core and in the others. Where Fusion or Cloud differ, the lesson says so.

Feature availability changes with each release. Check the release notes of your dbt version before depending on a newer feature, and pin versions in production.

Where dbt sits in a data platform

flowchart LR
  S["Apps and databases"] -->|"extract and load"| W[("Warehouse: raw schemas")]
  W -->|"dbt transforms"| M[("Warehouse: marts")]
  M --> B["BI tools"]
  M --> N["Notebooks and ML"]
  M --> R["Reverse ETL and apps"]

dbt reads raw schemas and writes marts inside the same warehouse.

Because dbt only issues SQL, the heavy computation always happens in the warehouse. A laptop can drive a transformation of billions of rows because it only sends text.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. In which stage are ref() calls turned into edges of the dependency graph?

    dbt discovers refs and sources while parsing and uses them to build the DAG before anything runs.
  2. 2. What does DAG stand for?

    A directed acyclic graph has one-way edges and no cycles, which makes a valid run order possible.
  3. 3. Which statement about dbt Core, Cloud and Fusion is correct?

    The project format (models, YAML, macros) is shared. The tools differ in hosting, scheduling and editor features.
  4. 4. Where does the heavy data processing of a dbt run happen?

    dbt only sends SQL. The warehouse executes it.
  5. 5. What happens when two models reference each other in a loop?

    A DAG cannot contain cycles, so dbt fails with an error naming the cycle.

Summary

  • A dbt command parses files, builds a DAG, compiles Jinja to SQL and then executes it in the warehouse.
  • Dependencies come only from ref() and source(); dbt derives order and parallelism from them.
  • Artifacts like manifest.json and run_results.json record what happened.
  • Core is the free CLI, Cloud is the hosted platform, Fusion is the Rust engine; all share the same project format.

Next: Project Structure and dbt_project.yml