Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 39 ยท Retention and Downsampling

Retention Periods and Data Expiry

Learn how InfluxDB expires old time-series data with retention periods, why whole shard groups are dropped, and how to choose a retention period per bucket.

Time-series data piles up fast. A single server reporting ten metrics every ten seconds produces about 86,000 points per day, and a fleet of a thousand servers produces tens of millions. Nobody wants to pay to keep every raw point forever, and nobody needs to: last year's per-second CPU reading is rarely useful, while last year's daily average is. Retention is how a time-series database forgets on purpose.

In this lesson you will learn what a retention period is, how InfluxDB enforces it by dropping whole blocks of data instead of deleting rows, and how to pick a sensible value. The InfluxDB server itself cannot run in your browser, so the examples use portable SQL and plain JavaScript stand-ins. Every stand-in is labelled, and every real InfluxDB command is shown as a read-only snippet.

What a retention period is

A retention period is a duration attached to a container of data. Data older than that duration, measured against the point's own timestamp and not against the time you wrote it, becomes eligible for removal.

  • In InfluxDB 2.x the container is a bucket. You choose a retention period when you create the bucket, for example 72h, 30d, or "forever" (a value of 0 means infinite).
  • In InfluxDB 1.x the container is a retention policy inside a database.
  • In InfluxDB 3 the container is a database, and you can set a retention period on it as well. Check the documentation of the edition you run, because the options differ between Core and Enterprise.

Creating a bucket with a 30-day retention period in InfluxDB 2.x looks like this (read-only; it needs a running server and a token):

Terminal
influx bucket create --name raw-metrics --retention 30d
influx bucket update --id 0a1b2c3d4e5f --retention 7d

Changing the period later is allowed. Shortening it makes old data eligible for removal soon after; lengthening it keeps data that has not yet been removed.

Retention is based on the timestamp inside the point. If you write a point stamped 40 days ago into a bucket that keeps 30 days, it is already expired, and InfluxDB may reject it or drop it soon after. Backfilling old data needs a bucket whose retention period covers it.

How data is actually removed

Deleting rows one by one is expensive: you would scan, mark, and rewrite storage. InfluxDB avoids that. It stores data in shards on disk, and each shard covers a fixed slice of time. Shards that cover the same slice are grouped into a shard group. The length of that slice is the shard group duration, and it is derived from the retention period by default: short retention gives short groups, long retention gives longer ones.

A shard group is dropped only when its entire time range lies beyond the retention period. That has a practical consequence: with a 3-day retention and 1-day shard groups, data can live a little longer than 3 days, because the group containing a slightly expired point still contains newer points. Expiry is coarse by design, and it makes cleanup almost free: removing a group is a file operation.

flowchart LR
  A["Shard group<br/>day 1"] --> B["Shard group<br/>day 2"] --> C["Shard group<br/>day 3"] --> D["Shard group<br/>day 4 (now)"]
  A -. "whole group older than retention: dropped" .-> X["Removed"]

Expiry removes whole shard groups, oldest first, once their end time passes the retention boundary.

InfluxDB 3 stores data as Parquet files partitioned by time, and the same idea applies: files that fall wholly outside the retention window are removed together.

Stand-in 1: retention as a delete in SQL

The closest portable idea is a DELETE with a cutoff. Real InfluxDB does not run SQL deletes for retention, but the effect on the data you can query is the same. The "current time" is fixed here so the result does not change each time you run it.

SQL Stand-in: enforce a 3-day retention window

Press Run (or Ctrl + Enter) to see the result here.

Notice that after the delete, the oldest remaining day has the full 48 points: the cutoff happens to fall on a day boundary. Change the cutoff to 2 * 86400 + 7200 and the oldest day shrinks to a partial day. Real InfluxDB would not behave like that, as the next example shows.

Stand-in 2: shard groups drop all at once

This simulation stores points in one-day groups and drops a group only when the group's end is older than the cutoff. Watch the "oldest surviving point" stay older than the retention period.

JS Stand-in: shard groups are dropped whole

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Choosing a retention period

Use these questions to pick a value:

  1. Who reads the data, and how far back? Live dashboards usually need hours to days of raw data. Capacity planning needs months of coarse data.
  2. What does it cost to keep? Storage grows linearly with retention for a steady write rate.
  3. Is there a legal or audit rule? Keep what you must, and keep it somewhere cheap.
  4. Will you downsample? If yes, keep raw data briefly (days) and the summarised data much longer (months or years). That is the topic of the next lesson.

A common layout is several buckets with different periods: raw for 7 days, hourly for 1 year, and daily forever.

Common mistakes

  • Setting a short retention period on a bucket and then being surprised that backfilled historical data disappears.
  • Expecting data to vanish at exactly the retention boundary. Whole groups go, so some old data lingers.
  • Forgetting that "infinite" retention on a busy bucket means unbounded disk growth.
  • Deleting by hand what retention would delete for free.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does InfluxDB compare against the retention period to decide whether a point has expired?

    Retention is evaluated against the point's own timestamp, not its write time.
  2. 2. Why can data live slightly longer than the retention period?

    Expiry removes whole shard groups, so a group that still holds some in-range points stays until its end passes the cutoff.
  3. 3. Which container carries the retention period in InfluxDB 2.x?

    In 2.x you set the retention period when creating or updating a bucket.
  4. 4. What is a sensible plan when you need both fine detail now and trends for years?

    Short raw retention plus long-lived summaries keeps costs low while preserving trends.

Summary

  • A retention period says how long data is kept, measured from each point's timestamp.
  • 2.x uses buckets, 1.x uses retention policies, 3.x sets it on the database.
  • Data is removed by dropping whole shard groups (or time-partitioned files), which is cheap but coarse.
  • Choose raw retention by what dashboards need, and keep summaries longer.
  • A value of 0 means keep forever, which means unbounded growth.

Next: Downsampling with Aggregate Windows