Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 38 ยท Streams, Tasks and Pipelines

Tasks and Task Graphs

Learn Snowflake tasks: schedules and cron, WHEN conditions with streams, task graphs (DAGs) with AFTER, serverless tasks, resuming, monitoring and error handling.

A task runs a piece of SQL (or a call to a stored procedure) on a schedule or after another task. Tasks turn separate statements into a pipeline: the stream says there is new data, the task processes it. Snowflake runs tasks for you, so there is no external scheduler to manage.

Tasks are a Snowflake feature that cannot run here. The statements are read-only. A Python stand-in models the scheduling and dependency logic so you can experiment.

Creating a task

SQL
CREATE TASK refresh_daily_sales
  WAREHOUSE = etl_wh
  SCHEDULE  = 'USING CRON 0 2 * * * UTC'      -- every day at 02:00 UTC
AS
  INSERT INTO daily_sales
  SELECT order_date, SUM(amount) FROM orders
  WHERE order_date = CURRENT_DATE() - 1
  GROUP BY order_date;

ALTER TASK refresh_daily_sales RESUME;        -- tasks are created suspended

Key points:

  • A task needs compute: either a named WAREHOUSE, or serverless compute when you omit it (Snowflake sizes and bills the compute per run; set USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE as a hint).
  • SCHEDULE is either an interval ('15 MINUTE', minimum 1 minute) or a cron expression with a time zone.
  • A new task is suspended. ALTER TASK ... RESUME starts it, SUSPEND stops it. Forgetting RESUME is the classic reason "my task never ran".
  • The role that runs it needs EXECUTE TASK on the account, and privileges on everything the SQL touches.
  • EXECUTE TASK name runs it once, immediately, which is ideal for testing.

Run only when there is work

Combine a short schedule with a WHEN condition. The condition is checked before the warehouse is started, so empty runs cost nothing:

SQL
CREATE TASK load_orders_clean
  WAREHOUSE = etl_wh
  SCHEDULE  = '1 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('orders_stream')
AS
  MERGE INTO orders_clean t
  USING orders_stream s ON t.id = s.id
  WHEN MATCHED THEN UPDATE SET amount = s.amount
  WHEN NOT MATCHED THEN INSERT (id, amount) VALUES (s.id, s.amount);

Task graphs

Real pipelines have steps. A task graph (also called a DAG) is a root task with a schedule plus child tasks that run AFTER their predecessors:

SQL
CREATE TASK extract   WAREHOUSE = etl_wh SCHEDULE = '60 MINUTE' AS CALL extract_proc();
CREATE TASK transform WAREHOUSE = etl_wh AFTER extract   AS CALL transform_proc();
CREATE TASK publish_a WAREHOUSE = etl_wh AFTER transform AS CALL publish_a_proc();
CREATE TASK publish_b WAREHOUSE = etl_wh AFTER transform AS CALL publish_b_proc();
CREATE TASK finalize  WAREHOUSE = etl_wh AFTER publish_a, publish_b AS CALL finalize_proc();
flowchart LR
  E["extract (root)<br/>every 60 min"] --> T["transform"]
  T --> A["publish_a"]
  T --> B["publish_b"]
  A --> F["finalize"]
  B --> F

Only the root has a schedule; children start when all their predecessors succeed.

Rules for graphs: one root, no cycles, up to 1,000 tasks per graph, resume children before the root (or SYSTEM$TASK_DEPENDENT_ENABLE for all), and suspend the root before changing the graph. Tasks can pass small values to each other with SYSTEM$SET_RETURN_VALUE and SYSTEM$GET_PREDECESSOR_RETURN_VALUE. A FINALIZE task can be attached to run once at the end whether the graph succeeded or failed, perfect for cleanup and alerts. Failed runs can be retried with TASK_AUTO_RETRY_ATTEMPTS.

A dependency scheduler

The Python module graphlib runs a topological sort, which is the heart of how a graph decides what can run. This stand-in shows the order and the parallel "waves".

Python Running a task graph (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

publish_a and publish_b run in the same wave, in parallel, and finalize waits for both. A graph with a cycle is refused, just as Snowflake refuses it.

Python Skip empty runs like a WHEN condition (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Serverless tasks and cost

Warehouse tasks charge for the warehouse while it runs; serverless tasks charge for the compute used per run (a multiplier applies) and need no warehouse. Short, frequent tasks that finish well under a minute usually suit serverless compute better, because a warehouse bills at least 60 seconds each time it resumes. Long heavy tasks run cheaper on a right-sized warehouse.

Monitoring and errors

SQL
SELECT name, state, scheduled_time, completed_time, error_message
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
       SCHEDULED_TIME_RANGE_START => DATEADD('hour', -24, CURRENT_TIMESTAMP())))
ORDER BY scheduled_time DESC;

SHOW TASKS;
SELECT * FROM TABLE(INFORMATION_SCHEMA.CURRENT_TASK_GRAPHS());

Set SUSPEND_TASK_AFTER_NUM_FAILURES so a broken task stops by itself, and attach error notifications (email, queue) to be told. Make task SQL idempotent (MERGE, delete-then-insert for a window) so a retry or overlap does no harm.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A task was created but never runs. What is the most likely cause?

    Tasks are created in a suspended state.
  2. 2. What does WHEN SYSTEM$STREAM_HAS_DATA('s') do?

    The condition is checked first, so empty runs cost nothing.
  3. 3. In a task graph, how does a child task start?

    Only the root has a schedule; the others use AFTER.
  4. 4. Which is NOT allowed in a task graph?

    A graph must be acyclic.
  5. 5. Serverless tasks are billed...

    Snowflake manages the compute and charges for what each run uses.

Summary

  • Tasks run SQL on a schedule or after other tasks; resume them first.
  • WHEN conditions with streams skip empty runs.
  • Task graphs have one root, AFTER dependencies and no cycles.
  • Serverless suits short frequent runs; monitor with TASK_HISTORY; make SQL idempotent.

Next: Dynamic tables