Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 27 ยท Production Practice

Scaling, Retention and Long-Term Storage

Size a Prometheus server, set retention, use federation and remote write, run HA pairs and compare Thanos, Mimir and VictoriaMetrics for long-term storage.

A single Prometheus server is more capable than most people expect: one well-sized machine can ingest millions of samples per second and hold millions of active series. But it is also, by design, a single node with local storage. It keeps a limited history, it is one machine that can fail, and it sees only the targets it scrapes. This lesson explains how to size it, how long to keep data, how the storage engine works, and what to reach for when one server is no longer enough: federation, remote write, high-availability pairs, and long-term storage systems such as Thanos, Mimir and VictoriaMetrics.

The guiding advice is to scale up first and out later. A bigger server and sensible cardinality are far simpler than a distributed system. Cardinality in particular (see Metrics, Labels and Time Series) is what actually decides how much memory and disk you need. The architecture background is in How Prometheus Works.

None of these systems can run on this site, so configuration is read-only. The runnable examples are JavaScript stand-ins: a capacity calculator and a downsampler, which show the arithmetic and the idea behind long-term storage. The constants are rough rules of thumb, not guarantees; measure your own server.

When one server is not enough

History Local disk keeps weeks, not years.
Failure One machine is a single point of failure.
Reach Many clusters need one global view.
1 / 2

Sizing a server

Three numbers drive the cost of a Prometheus server:

  • Active series: how many distinct time series have received samples recently. This drives memory, because the newest data of every series is kept in RAM, at a cost of roughly a few kilobytes per series.
  • Ingestion rate: series divided by the scrape interval, in samples per second. This drives CPU and disk writes.
  • Retention: how long data is kept. This drives disk. Compressed samples take roughly one to two bytes each, so disk is about retention seconds x samples per second x bytes per sample.

For example, 500,000 series scraped every 15 seconds is about 33,000 samples per second; kept for 15 days at 1.5 bytes per sample that comes to around 65 GB. Add headroom for compaction, which temporarily needs extra space, and for query memory, since an expensive query can use more RAM than ingestion does.

JS A capacity calculator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last scenario shows the usual cause of an outage: nobody planned for 10 million series, one label with many values appeared, and memory ran out. The same arithmetic also shows why simply keeping one year of full-resolution data on a local disk gets expensive.

Retention

Retention is set with startup flags, not in prometheus.yml:

Terminal
prometheus \
  --config.file=prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=200GB

Data is dropped when it is older than the time limit or when the blocks exceed the size limit, whichever comes first, and the default time retention is 15 days. Choose a size limit below the actual disk (leave 10 to 20 percent free) so compaction never fails on a full disk. Local retention is meant for recent, detailed data (days to weeks). For months or years, you move the data elsewhere (see below).

How the storage works (TSDB internals)

Understanding the layout explains most operational advice:

  • Incoming samples go to the head block, held in memory, and are also appended to a write-ahead log (WAL) on disk. After a crash, Prometheus replays the WAL to recover; this is why a restart with a big head can take minutes.
  • About every two hours the head is written to disk as an immutable block: a directory with an index (label values to series), chunk files holding compressed samples, and metadata.
  • Background compaction merges small blocks into larger ones, and retention deletes whole blocks that have aged out. Because blocks are immutable, deleting data means deleting a block, which is why retention is coarse.
  • Queries read the head and the blocks that overlap the requested range, so a query over a long range touches many blocks.

Samples are compressed using their regular structure (timestamps are stored as small deltas of deltas, values with XOR encoding), which is how a sample fits in about one to two bytes. Highly irregular or random values compress worse.

Two practical rules follow. Put the data directory on fast local disk (SSD); network file systems such as NFS are not supported for the TSDB. And treat a Prometheus server's data as replaceable: it is a time-limited cache of recent metrics, not an archive, so back it up only if you truly need that exact history (the snapshot API and the remote-write path exist for that).

High availability

Because one server is one machine, the standard approach is to run two identical Prometheus servers that scrape the same targets independently. If one dies, the other still has the data and keeps evaluating alerts. They are not clustered; each has slightly different timestamps and their own copy.

Give each a distinguishing external label, so downstream systems can tell the copies apart (and deduplicate them):

YAML
global:
  scrape_interval: 15s
  external_labels:
    cluster: eu-1
    replica: a   # "b" on the second server

Point both at all Alertmanager instances, which deduplicate the identical alerts (see Alerting Rules and Alertmanager). For dashboards, query either server, or a layer that merges and deduplicates both, such as Thanos Query.

Scaling out: sharding and federation

When one server truly cannot hold all series, split the work. Functional sharding gives each team, service or datacenter its own Prometheus, which is the simplest approach and often matches ownership anyway. Hash-based sharding divides targets of one huge job between servers using relabeling (hashmod), as covered in Service Discovery and Relabeling.

To get a combined view, federation lets one Prometheus scrape selected series from another's /federate endpoint:

YAML
scrape_configs:
  - job_name: federate
    honor_labels: true
    metrics_path: /federate
    params:
      "match[]":
        - '{__name__=~"job:.*"}'   # only pre-aggregated recording rules
    static_configs:
      - targets: ["prom-eu-1:9090", "prom-us-1:9090"]

Federate aggregated series (the level:metric:operations recording rules from Recording Rules and Query Performance), not raw ones; pulling everything just moves the cardinality problem to the top server. Federation also is not a long-term-storage solution: the top-level server has the same local retention limits.

Remote write and long-term storage

Remote write makes Prometheus stream every sample it ingests (or a filtered subset) to another system over HTTP, in addition to storing it locally:

YAML
remote_write:
  - url: https://metrics.example.com/api/v1/write
    queue_config:
      max_samples_per_send: 5000
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*|process_.*"
        action: drop

A long-term storage system receives that stream (or reads Prometheus blocks) and keeps the data cheaply for months or years in object storage, with a global query layer. The main families:

  • Thanos adds a sidecar next to each Prometheus that uploads its blocks to object storage (S3, GCS, Azure), plus components to query across servers and deduplicate HA pairs, and a compactor that can downsample old data.
  • Grafana Mimir (and its relative Cortex) is a horizontally scalable, multi-tenant backend that receives data by remote write and stores it in object storage.
  • VictoriaMetrics is a Prometheus-compatible time series database, available as a single binary or as a cluster, known for low resource use; it accepts remote write and can also scrape.

All three speak PromQL (VictoriaMetrics with a superset called MetricsQL), so your Grafana dashboards keep working. The choice depends on operational taste: sidecars and object storage (Thanos), a multi-tenant service you scale by adding components (Mimir), or a compact database that is simple to run (VictoriaMetrics). Check each project's current documentation and release notes before deciding; they evolve quickly.

Downsampling

Nobody needs 15-second resolution from last year. Downsampling replaces old raw samples with fewer aggregated points (for example, one point per 5 minutes, then one per hour), which cuts storage and makes long-range queries fast. Typically each bucket stores several values (minimum, maximum, sum, count, and sometimes the last counter value), so averages and rates can still be computed.

JS Downsampling a series

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last two lines are the main design point: if downsampling kept only the average, the 5-minute incident would almost disappear from the hourly view. That is why systems store min, max, sum and count per bucket.

Choosing a path

  1. Start with one server, 15 to 30 days of retention and recording rules for expensive queries.
  2. Add a second replica when the monitoring itself must survive failures.
  3. Add remote write to a long-term store when you need months of history, a global view across clusters, or durable storage off the machine.
  4. Shard or federate only when a single server cannot hold the active series, and keep cardinality under control throughout.
  5. Monitor Prometheus itself: prometheus_tsdb_head_series, process_resident_memory_bytes, prometheus_remote_storage_samples_pending and rule evaluation durations.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which resource is mostly driven by the number of active series?

    Active series determine head-block memory. Retention mostly determines disk, and ingestion rate drives CPU.
  2. 2. How is retention configured in Prometheus?

    Retention is a command-line flag. Whole blocks are deleted when they exceed the time or size limit.
  3. 3. What is the purpose of the write-ahead log (WAL)?

    Samples in the head block are also appended to the WAL, which Prometheus replays on startup.
  4. 4. What should you federate to a global Prometheus?

    Federating raw series moves the cardinality problem to the top server. Aggregated recording-rule series keep it small.
  5. 5. Why does downsampling store min, max, sum and count per bucket instead of only an average?

    An average alone would hide a short incident in a long bucket. Keeping the extremes and counts preserves the information that dashboards need.

Summary

  • Memory scales with active series, CPU with ingestion rate and disk with retention; size by series x retention x bytes per sample and watch cardinality first.
  • Retention is set with flags (retention.time and retention.size); local storage is for recent, detailed data.
  • The TSDB keeps recent data in a memory head backed by a WAL, writes immutable blocks about every two hours and compacts them.
  • Run two identical replicas with distinct external labels for high availability, and let Alertmanager deduplicate.
  • Federate aggregated series, use remote write to feed Thanos, Mimir or VictoriaMetrics for long-term storage, and downsample old data while keeping min, max, sum and count.

Next: Monitoring Kubernetes and Containers