Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 43 ยท Working from Code

Logstash and Beats for Data Ingestion

Collect and ship data to Elasticsearch with Filebeat, Elastic Agent and Logstash: input, filter and output plugins, conditionals, multiple pipelines and when to pick which tool.

Elasticsearch stores and searches documents, but something has to bring those documents in. Log files sit on hundreds of servers, orders live in a database, events flow through Kafka. The Elastic Stack has two families of tools for this job: lightweight shippers (Beats and Elastic Agent) that run next to the data, and Logstash, a heavier, central processing engine.

This lesson explains what each tool is for, shows a real Logstash configuration, and lets you run a stand-in that behaves like a Logstash pipeline. None of the real tools run on this site, so the configuration files are read-only and the stand-in is plain JavaScript. It imitates the shape of a pipeline (inputs, filters, outputs), not Logstash's code.

Two kinds of ingest tools

Beats and Agent Small, run on each host.
Logstash Central, heavy transforms.
Elasticsearch Stores and indexes the result.

Shippers collect. Logstash reshapes. Elasticsearch searches.

1 / 2

Beats and Elastic Agent

A Beat is a small single-purpose program written in Go. Filebeat tails log files, Metricbeat reads system and service metrics, Packetbeat watches network traffic, and others handle audit data or uptime checks. Each one reads data, adds metadata such as host name and timestamp, remembers how far it got (so a restart does not duplicate or skip lines), and sends events onward.

Elastic Agent is the newer, unified shipper. One agent on a host can collect logs, metrics and security data, and you manage its policies centrally from Kibana through Fleet instead of editing files on every machine. For new deployments Elastic Agent is the usual starting point, and Beats are still widely used and fully supported.

A minimal Filebeat configuration:

YAML
filebeat.inputs:
  - type: filestream
    id: web-access
    paths:
      - /var/log/nginx/access.log
    fields:
      service: web

output.elasticsearch:
  hosts: ["https://es.example.com:9200"]
  api_key: "id:api_key_value"
  pipeline: web-logs

Filebeat sends lines straight to Elasticsearch and asks it to run the web-logs ingest pipeline. That is the simplest design: shippers on hosts, parsing in Elasticsearch, no extra servers. It is a good fit when parsing is light.

Logstash

Logstash is a server-side data processing engine that runs on the JVM. It reads from many sources, transforms events with a large set of plugins, and writes to many destinations. Reach for it when you need:

  • Many source types, such as relational databases (jdbc), message queues (kafka, rabbitmq), HTTP endpoints or cloud storage.
  • Heavy or complex transformation that you do not want to run on ingest nodes.
  • Multiple outputs, for example the same event to Elasticsearch and to object storage.
  • Buffering: a persistent queue on disk absorbs bursts so a slow cluster does not lose events.

A pipeline configuration

A Logstash pipeline file has up to three sections. Plugins in each section run in the order written.

TEXT
input {
  beats {
    port => 5044
  }
  kafka {
    bootstrap_servers => "kafka1:9092"
    topics => ["app-logs"]
    codec => "json"
  }
}

filter {
  if [service] == "web" {
    grok {
      match => { "message" => "%{IPORHOST:client_ip} %{WORD:method} %{URIPATH:path} %{NUMBER:status:int}" }
    }
    date {
      match => ["timestamp", "ISO8601"]
      target => "@timestamp"
    }
    geoip { source => "client_ip" target => "client_geo" }
  } else {
    mutate { add_tag => ["unparsed"] }
  }
  mutate {
    remove_field => ["timestamp"]
    lowercase => ["method"]
  }
}

output {
  elasticsearch {
    hosts => ["https://es.example.com:9200"]
    api_key => "${ES_API_KEY}"
    index => "app-logs-%{+YYYY.MM.dd}"
  }
  if "unparsed" in [tags] {
    file { path => "/var/log/logstash/unparsed.log" }
  }
}

Points worth knowing:

  • Events are the unit Logstash handles. Field references use square brackets: [service], [client][geo].
  • Conditionals (if, else if, else, in, and, or) work in filter and output sections.
  • grok, date, geoip, mutate do the same jobs as the processors of the same names in ingest pipelines. mutate bundles many small edits (rename, convert, split, remove). Others include dissect, json, kv, csv, drop and ruby for custom code.
  • Codecs such as json decode data as it enters or leaves a plugin.
  • ${ES_API_KEY} reads a value from the Logstash keystore or environment, so secrets stay out of the file.

Several pipelines in one Logstash

One Logstash process can run several independent pipelines, each with its own workers and queue. Declare them in pipelines.yml:

YAML
- pipeline.id: web
  path.config: "/etc/logstash/conf.d/web.conf"
  pipeline.workers: 4
- pipeline.id: orders-db
  path.config: "/etc/logstash/conf.d/orders.conf"
  queue.type: persisted

Separate pipelines stop a slow database job from blocking the web logs. When you do want to chain them, use the pipeline input and output plugins to send events from one pipeline to another inside the same process.

Reading from a database

The jdbc input runs a SQL query on a schedule and turns each row into an event. Tracking a column such as updated_at lets each run fetch only new or changed rows:

TEXT
input {
  jdbc {
    jdbc_connection_string => "jdbc:postgresql://db:5432/shop"
    jdbc_user => "reader"
    jdbc_driver_class => "org.postgresql.Driver"
    schedule => "*/5 * * * *"
    statement => "SELECT * FROM products WHERE updated_at > :sql_last_value"
    use_column_value => true
    tracking_column => "updated_at"
    tracking_column_type => "timestamp"
  }
}

Polling like this does not notice deleted rows. For deletes you need soft-delete flags or a change data capture approach (see the Kafka change data capture project for the idea).

Choosing between them

Situation Good choice
Tail log files on many servers Filebeat or Elastic Agent
Light parsing, Elasticsearch can do it Shipper plus an ingest pipeline
Many source types, databases, queues Logstash
Heavy transformation, several outputs Logstash
Centrally managed fleet of hosts Elastic Agent with Fleet
Your own application writes events The client directly, see clients and bulk

Setups often combine them: Beats on every host send to Logstash, which parses and forwards to Elasticsearch. Back-pressure matters in that chain: when Elasticsearch is slow, Logstash's queue fills, and the shippers slow down instead of dropping data.

A Logstash-style pipeline you can run

The stand-in below reads a few events from two inputs, runs filters with conditionals, and routes events to outputs. It shows how conditionals decide what happens to each event.

JS Input, filter, output with conditionals

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Buffering and back-pressure

Logstash holds events in an in-memory queue by default. Setting queue.type: persisted stores them on disk, so a crash or a long Elasticsearch outage does not lose what is queued. A dead letter queue can keep events that Elasticsearch rejected, for example because of a mapping conflict, so you can fix and replay them.

This stand-in models a bounded queue: the producer pauses when the queue is full, which is what back-pressure means.

JS A bounded queue and back-pressure

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which tool is best for tailing log files on many servers with little overhead?

    Beats and Elastic Agent are lightweight shippers made to run next to the data.
  2. 2. Which Logstash feature lets one process run independent flows so a slow one does not block another?

    Each pipeline has its own workers and queue, so they are isolated.
  3. 3. A jdbc input with an updated_at tracking column will NOT notice which change?

    A query can only return rows that still exist, so deletions need another mechanism.
  4. 4. What does a persistent queue protect against?

    queue.type: persisted writes buffered events to disk.

Summary

  • Beats and Elastic Agent are lightweight shippers; Elastic Agent adds central management through Fleet.
  • Logstash is a central engine with input, filter and output plugins, conditionals and many source and destination types.
  • A shipper plus an ingest pipeline is the simplest design; use Logstash for varied sources, heavy transforms or several outputs.
  • Run several pipelines in one Logstash to isolate workloads, and use persistent and dead letter queues for safety.
  • Database polling with jdbc fetches changes by a tracking column but cannot see deletes.

Next: see what you have indexed with the Kibana tour.