Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 39 ยท Retention and Downsampling

Scheduled Downsampling with Tasks

Automate downsampling with InfluxDB tasks, understand every and offset, and make rollups safe to re-run with idempotent writes. Includes runnable stand-ins.

Running a rollup query by hand once is easy. Running it correctly every hour for years is the real job. InfluxDB 2.x solves this with tasks: Flux scripts that the server runs on a schedule. A downsampling task reads the last window of raw data, aggregates it, and writes the result into a long-retention bucket.

In this lesson you will read a real task, learn what every and offset do, and see why a rollup must be idempotent (safe to run twice). The scheduler itself needs a running server, so the real task is shown read-only and the behaviour is demonstrated by a JavaScript scheduler and an SQL upsert.

Anatomy of a downsampling task

This is a typical InfluxDB 2.x task (read-only):

JavaScript
option task = {name: "downsample-cpu-hourly", every: 1h, offset: 5m}

from(bucket: "raw")
  |> range(start: -task.every)
  |> filter(fn: (r) => r._measurement == "cpu")
  |> aggregateWindow(every: 1h, fn: mean, createEmpty: false)
  |> to(bucket: "hourly", org: "my-org")

Walk through it line by line:

  • option task declares the schedule. every: 1h runs once per hour; you can use cron instead for a calendar schedule such as "0 * * * *".
  • offset: 5m delays each run by five minutes after the boundary, giving late-arriving points time to land.
  • range(start: -task.every) reads the window since the previous run.
  • aggregateWindow creates the summary rows.
  • to(bucket: ...) writes them to the destination bucket.

You create it with influx task create --file downsample.flux, or in the UI under Tasks. Each execution is a run with a status (success, failed, canceled) and logs. Failed runs can be retried.

In InfluxDB 3 there is no Flux. You schedule downsampling with an external job (cron, a workflow tool) or with the server's processing engine, and write the result back with a SQL query result and the write API. The principle is the same.

flowchart LR
  S["Scheduler<br/>every 1h + offset 5m"] --> Q["Read last window<br/>from raw bucket"]
  Q --> A["Aggregate<br/>mean, min, max, count"]
  A --> W["Write summaries<br/>to hourly bucket"]

A downsampling task is a read, an aggregate, and a write on a timer.

Why offset matters

Points often arrive late: network delays, batching in agents, devices that reconnect. If the 10:00 run processes the window 09:00 to 10:00 at exactly 10:00:00, points that carry a 09:59:58 timestamp but arrive at 10:00:03 are missed, and the summary is permanently too small. An offset of one to five minutes costs nothing and removes most of that risk.

The stand-in below makes this visible. Points have an arrival delay; a run with no offset sees fewer points than a run with a 5-minute offset.

JS Stand-in: late points and the offset

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Choose the offset from your real arrival delays: look at the difference between event time and write time under normal load, and add a margin.

Idempotent rollups

Schedulers fail and retry. A run may crash after writing half of its output, then run again. If rollup rows are appended blindly, you get duplicates and inflated sums.

InfluxDB has a built-in answer: a point is identified by measurement + tag set + timestamp, and writing a point with the same identity overwrites the field values. So a re-run that writes the same window produces the same rows and no duplicates, as long as your window timestamps are deterministic (aligned to the window start, not to "now").

In SQL, the same effect uses a primary key and an upsert:

SQL Stand-in: re-running a rollup twice is safe

Press Run (or Ctrl + Enter) to see the result here.

Two runs, two rows. The key (window_start, host) plays the role of InfluxDB's point identity.

A tiny task scheduler

To internalise how a task covers time without gaps or overlaps, run this loop. Each run handles exactly the window that ended at its boundary, and it records which windows were done so a restart does not repeat or skip any.

JS Stand-in: a scheduler that never skips a window

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Real InfluxDB tasks offer similar catch-up behaviour through the latestCompleted time and manual backfill runs for a past range. After an outage, check the task's run list and trigger runs for any windows that were missed.

Cascading roll-ups

Chain levels: raw to 1 minute to 1 hour to 1 day. Each task reads the level below it, so it processes far fewer points. This is where the lesson of aggregate windows pays off: store count and sum so each level can combine the previous one exactly.

Common mistakes

  • No offset, so late points are lost from summaries.
  • Windows aligned to "now" rather than to boundaries, which breaks idempotency.
  • Writing summaries into the same bucket and measurement as raw data, mixing resolutions in one query result.
  • Setting the destination bucket's retention shorter than the source's.
  • Never monitoring the task itself. Alert on failed runs (see checks and notifications).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does offset do in an InfluxDB task option?

    Offset delays execution, giving late-arriving points time to be written before the window is summarised.
  2. 2. Why is it safe to re-run a rollup that writes points with deterministic timestamps?

    Point identity is measurement plus tag set plus timestamp, so a repeated write replaces rather than duplicates.
  3. 3. Which Flux function writes task output into another bucket?

    to() writes the stream's rows to a bucket.
  4. 4. InfluxDB 3 has no Flux. How do you schedule downsampling there?

    You run a SQL aggregate on a schedule from outside (or from the server's plugin engine) and write the summaries back.

Summary

  • A task is a scheduled Flux script: read the last window, aggregate, write to another bucket.
  • Use offset so late points are included.
  • Align window timestamps to boundaries so repeated runs overwrite instead of duplicating.
  • Track completed windows and backfill after outages.
  • Chain levels (minute, hour, day), storing count and sum.

Next: Building Dashboards