Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 39 ยท Integrations

Telegraf Pipelines: Inputs to Outputs

Build Telegraf pipelines for InfluxDB with inputs, processors, aggregators and outputs. Read real TOML configs and run a JavaScript model of the pipeline.

Telegraf is InfluxData's open-source agent for collecting and shipping metrics. It is a single binary written in Go with hundreds of plugins, so you rarely write collection code yourself: you describe what to read, how to reshape it and where to send it in a TOML file. It is the standard bridge between your servers, devices and services and InfluxDB (and many other destinations).

This lesson explains the four plugin types, walks through a real configuration (read-only, since the agent cannot run here), and then runs a small JavaScript model of the pipeline so you can see how a metric changes at each stage.

The four stages

flowchart LR
  I["Inputs<br/>cpu, mem, mqtt, http"] --> P["Processors<br/>rename, convert, filter"]
  P --> A["Aggregators<br/>mean, min, max per period"]
  A --> O["Outputs<br/>influxdb_v2, file, kafka"]
  P --> O

Metrics flow from inputs through optional processors and aggregators to one or more outputs.

  1. Input plugins collect metrics, either by polling (every interval) or by listening (an HTTP or MQTT listener).
  2. Processor plugins change metrics one at a time: rename, add or remove tags, convert types, parse strings, drop fields.
  3. Aggregator plugins combine metrics over a period (for example one-minute mean) and emit the result.
  4. Output plugins write the final metrics to a destination.

Every metric Telegraf handles has the same shape as an InfluxDB point: a name (measurement), tags, fields and a timestamp. That shared model is why Telegraf and InfluxDB fit together so well.

A complete configuration

This read-only example collects CPU, memory and disk, adds a tag, keeps one-minute aggregates and writes to InfluxDB 2.x:

TOML
[agent]
  interval = "10s"
  round_interval = true
  flush_interval = "10s"
  metric_batch_size = 5000
  metric_buffer_limit = 50000
  omit_hostname = false

[global_tags]
  environment = "production"
  datacenter = "eu-west"

[[inputs.cpu]]
  percpu = false
  totalcpu = true

[[inputs.mem]]

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "overlay"]

[[processors.rename]]
  [[processors.rename.replace]]
    measurement = "mem"
    dest = "memory"

[[aggregators.basicstats]]
  period = "60s"
  drop_original = false
  stats = ["mean", "max"]
  namepass = ["cpu"]

[[outputs.influxdb_v2]]
  urls = ["https://influx.example.com:8086"]
  token = "${INFLUX_TOKEN}"
  organization = "my-org"
  bucket = "raw"

What each section does:

  • [agent] sets the collection interval, how often to flush to outputs, the batch size and the in-memory buffer (metric_buffer_limit), which absorbs outages.
  • [global_tags] attaches tags to every metric, so you can filter by environment.
  • [[inputs.*]] blocks read data. Double brackets mean "a list of plugin instances", so you can repeat a plugin with different settings.
  • [[processors.*]] and [[aggregators.*]] transform data.
  • namepass, namedrop, tagpass, fieldpass are metric filters available on any plugin, to choose which metrics it applies to.
  • ${INFLUX_TOKEN} reads an environment variable so the secret stays out of the file.

For InfluxDB 3, the output plugin is outputs.influxdb_v2 pointed at the v2-compatible write endpoint, or a dedicated plugin where available; check the documentation of your version for the current plugin and options.

Useful commands (read-only):

Terminal
telegraf --config telegraf.conf --test        # run once, print metrics, send nothing
telegraf config > telegraf.conf               # generate a sample file
telegraf --config telegraf.conf --input-filter cpu:mem --output-filter influxdb_v2 config

Always run --test before deploying a changed config.

Stand-in: a model of the pipeline

This JavaScript model reproduces the stages on sample metrics: global tags, a rename processor with a filter, and a basicstats-like aggregator. It is a teaching model, not Telegraf.

JS Stand-in: inputs, processor, aggregator, output

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The output contains the original cpu points, the renamed memory point and the aggregate (drop_original = false keeps the originals). Set drop_original = true in the real config to send only the aggregate.

Processors worth knowing

Plugin Use
processors.rename Rename measurements, tags or fields
processors.converter Convert tags to fields or integers to floats
processors.regex Rewrite tag or field values with regular expressions
processors.override Set or override measurement name and tags
processors.strings Trim, lowercase, replace
processors.enum Map codes to names (1 to ok)
processors.dedup Drop unchanged values to save bandwidth
processors.starlark Custom logic in a small Python-like language

Processors run in the order set by their order option, so be explicit when more than one touches the same metrics.

Operating Telegraf

  • Run as a service (systemd, Windows service, or a container) with a config in /etc/telegraf.
  • Split config into files under /etc/telegraf/telegraf.d/, one per input, so teams own their pieces.
  • Watch for dropped metrics. The internal input plugin reports gathered, dropped and written counts: graph them.
  • Set metric_buffer_limit to cover the outages you expect. A full buffer drops the oldest metrics.
  • Secrets come from environment variables or a secret store, not plain text.
  • Use --test and --once while developing.

Common mistakes

  • Intervals shorter than the real collection time, leading to overlapping gathers.
  • Per-container or per-process inputs without filters, creating a cardinality explosion.
  • Tokens in the config file committed to version control.
  • Many outputs without understanding that a slow output can back up the buffer.
  • Forgetting namepass, so an aggregator or processor changes metrics you did not intend.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In what order do the main Telegraf stages run?

    Metrics are collected by inputs, transformed by processors, optionally aggregated, and written by outputs.
  2. 2. What does namepass = ["cpu"] on a plugin do?

    Metric filters such as namepass limit which metrics a plugin applies to.
  3. 3. Which command runs the configured inputs once and prints metrics without writing them?

    --test gathers and prints metrics without sending them to outputs.
  4. 4. How should the InfluxDB token reach the config?

    Environment variables keep secrets out of files under version control.

Summary

  • Telegraf has inputs, processors, aggregators and outputs, configured in TOML.
  • Metrics share the InfluxDB point shape: name, tags, fields, timestamp.
  • Filters (namepass, tagpass) control where each plugin applies.
  • Test with --test, keep tokens in env vars, monitor the internal input.
  • Watch cardinality introduced by inputs.

Next: Client Library Patterns