Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 37 ยท Logs

Datadog Log Indexes, Exclusion Filters, Archives and Log-Based Metrics

Control log cost and retention in Datadog: ingestion versus indexing, indexes with retention, exclusion filters, archives and rehydration, generate metrics from logs, and sensitive data scanning.

Logs are the largest line on many Datadog invoices, and the reason is simple: volume. A busy service can emit gigabytes per hour. Datadog gives you several layers to decide which logs to keep searchable, which to store cheaply, and which to only count. Understanding the layers lets you keep every log you might need while paying for far fewer of them.

Datadog does not run on this site, so settings are described in words and the stand-ins model the arithmetic and the routing.

Ingest, index, archive

Three separate things can happen to a log:

  1. Ingested: it reached Datadog. You are billed per gigabyte ingested, and everything ingested passes through pipelines, Live Tail, log-based metrics, security rules and archives.
  2. Indexed: it was stored in an index so it is searchable in the Log Explorer, usable in dashboards and monitors. Indexing is billed per million events, per retention period.
  3. Archived: a copy was written to your own cloud storage (S3, Azure Blob, Google Cloud Storage) for long, cheap retention.

Not everything needs to be indexed. A log can be ingested and archived without ever being indexed, and metrics can still be generated from it.

Indexes and retention

An index is a logical bucket with:

  • A filter (a log search query) deciding which logs go in it.
  • A retention period (for example 3, 7, 15, 30 days; available values depend on your plan).
  • An optional daily quota with a warning threshold.
  • A list of exclusion filters.

You can have several indexes (for example prod-main with 15 days and debug with 3 days). Logs go to the first index whose filter matches, in the order shown on the page, so put narrow filters first. Access to an index can be limited by role.

Exclusion filters

An exclusion filter drops logs from an index (they are not stored or billed as indexed) while they stay ingested. You can drop by query and by a sampling rate:

TEXT
Filter:   service:storefront status:info
Sampling: exclude 90% of matching logs

That keeps 10% of info logs as a representative sample. Typical targets: health checks, debug logs, successful requests with 2xx status. Always keep 100% of errors and warnings and preserve everything for the important services.

Because exclusion drops logs from the index, what you see in the explorer is no longer a full count. That is why you pair exclusion with a metric (below) when you still need totals.

Newer plans also offer Flex Logs, a cheaper storage tier for long retention that you can still query (with different performance and pricing). Check current plan details before relying on it.

Archives and rehydration

An archive is a rule that writes logs, filtered by a query, to your storage bucket in compressed JSON files. Setup outline:

  1. Create the bucket and a role/permission so Datadog can write to it.
  2. In Logs > Configuration > Archives, add an archive: destination, path prefix and a filter (often * to keep everything).
  3. Logs flow in continuously, whether or not they were indexed or excluded.

If you later need old logs, rehydrate (Historical View): select an archive, a time window and a query, and Datadog reads the files and re-indexes those logs into a temporary index so you can search them. Rehydration takes time and costs indexing for what you bring back, so scope it narrowly. Archives are also a compliance tool, since the data lives in your account under your encryption keys.

Generate metrics from logs

A log-based metric counts logs matching a query, or aggregates a measure (such as request duration), and emits a regular Datadog metric. It is computed on all ingested logs, even those excluded from indexes, and the resulting metric is retained like any metric (15 months).

Create it in Logs > Generate Metrics:

  • Query: service:checkout @http.status_code:>=500
  • Type: count, or distribution of a measure like @duration (with percentiles if you enable them)
  • Group by: tags such as env, @http.route (keep these low-cardinality, because the metric becomes a custom metric billed per series)
  • Name: shop.checkout.http_errors

Use log-based metrics for alerts and dashboards when you do not need the log lines themselves: total requests, errors by route, latency percentiles. You can then exclude most of those logs from indexing at no loss of insight.

Protect sensitive data

The Sensitive Data Scanner looks for patterns (credit card numbers, API keys, emails, tokens) in logs, spans and RUM events and can redact, hash or partially mask them before storage. It applies in the Datadog backend; the Agent's mask_sequences (see log collection) applies earlier on the host. Use both for defense in depth, and limit who can read logs with roles.

A cost-aware design

flowchart LR
    A["All logs<br/>(ingested)"] --> P["Pipelines"]
    P --> M["Log-based metrics<br/>(count everything)"]
    P --> AR["Archive to S3<br/>(cheap, all logs)"]
    P --> I1["Index: errors<br/>30 days, 100%"]
    P --> I2["Index: info<br/>3 days, 10% sample"]

Count everything, archive everything, index only what you search.

A common recipe:

  1. Archive everything to cheap storage.
  2. Keep errors and warnings indexed for 15โ€“30 days.
  3. Sample or exclude info/debug/health-check logs.
  4. Generate metrics for counts and latency from the unsampled stream.
  5. Set a daily quota so a runaway deploy cannot produce a surprise bill.
  6. Review Plan & Usage weekly for the largest sources, and fix noisy loggers at the source.
Python Stand-in: how exclusion and sampling change indexed volume

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: route logs to the first matching index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The route is decided by order, so swapping the first two indexes would send errors to the 7-day index. Keep narrow, important filters first.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the difference between ingesting and indexing a log?

    Ingestion is arrival (billed per GB); indexing makes logs searchable (billed per million events).
  2. 2. Why create a log-based metric before excluding most info logs?

    Log-based metrics are computed on ingested logs, so totals remain accurate after exclusion.
  3. 3. What is rehydration?

    Rehydration reads archive files and brings matching logs back into a searchable index.
  4. 4. Logs match several indexes' filters. Where do they go?

    Index order matters; the first match wins.
  5. 5. Which settings protect against a surprise bill from a noisy deploy?

    Daily quotas cap indexing; combine them with exclusion filters and alerts on usage.

Summary

  • Ingestion, indexing and archiving are separate stages, billed separately.
  • Indexes combine a filter, retention, optional quota and exclusion filters; first match wins.
  • Archives keep everything cheaply in your own storage, and rehydration brings logs back.
  • Log-based metrics count all ingested logs; use them to replace expensive indexed searches.
  • Scan and mask sensitive data, and set quotas and usage alerts.

Next: APM Overview