Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 37 ยท Logs

Datadog Log Collection: Tail Files, Containers, HTTP and Processing Rules

Collect logs with Datadog: enable logs_enabled, tail files with conf.yaml, container logs, HTTP intake, multi-line rules, exclude and mask rules, source and service, and file permissions.

Metrics tell you that something is wrong; logs tell you why. Datadog Log Management collects logs from files, containers, cloud services and applications, parses them into structured fields, and lets you search them next to your metrics and traces. This lesson covers the first half: getting logs in. The next lessons cover parsing, searching and retention and cost.

The Agent and the product do not run on this site. Configuration is shown read-only, and Python stand-ins model the rules.

Turn on log collection

Logs are off by default. Enable them in datadog.yaml (or DD_LOGS_ENABLED=true):

YAML
logs_enabled: true

Restart the Agent, then tell it what to read.

Tail a file

Create a folder for the source under conf.d and a conf.yaml with a logs: section:

YAML
# /etc/datadog-agent/conf.d/storefront.d/conf.yaml
logs:
  - type: file
    path: /var/log/storefront/app.log
    service: storefront
    source: nodejs
    tags:
      - env:prod
      - team:web

The fields that matter:

  • type: file tails a file (wildcards like /var/log/app/*.log work).
  • service names the application; it should match the service tag on its metrics and traces.
  • source names the technology (nginx, postgresql, java, nodejs). A known source triggers an integration pipeline that parses that software's log format automatically.
  • tags add context.

The Agent runs as dd-agent, so that user needs read permission on the file and execute on its folders. datadog-agent status shows a "Logs Agent" section that lists each source and says OK (with bytes read) or shows an error such as permission denied. Rotation (logrotate, copytruncate or rename) is followed automatically.

Other sources

Source How
TCP / UDP type: tcp or udp with a port, for apps that write to a socket or syslog
journald type: journald, optionally with include_units
Windows Event Log type: windows_event with channel_path such as System or Application
Docker Set DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true, or per-container labels/annotations
Kubernetes Helm datadog.logs.containerCollectAll: true; per-pod ad.datadoghq.com/<container>.logs annotation
Cloud services Datadog Forwarder (Lambda), Firehose, Event Hub or Pub/Sub, see cloud integrations
HTTP API POST JSON to the log intake with your API key
Serverless Lambda extension, Cloud Run sidecar

For containers, the Agent reads the runtime's log files and tags the logs with container, pod and image tags automatically. Applications should log to standard output and standard error; the platform stores those streams and the Agent picks them up. Avoid writing log files inside containers.

Sending logs over HTTP

When you cannot run an Agent (a script, an edge service), send JSON directly:

Terminal
curl -X POST "https://http-intake.logs.datadoghq.com/api/v2/logs" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -d '[{"message":"payment failed","ddsource":"python","service":"billing",
        "ddtags":"env:prod,team:payments","status":"error","order_id":"A-1001"}]'

ddsource, service, hostname and ddtags are reserved fields. Use the intake domain for your site (for example http-intake.logs.datadoghq.eu). Batch many events per request instead of sending one at a time.

Multi-line logs

A stack trace spans several lines but is one event. Without help the Agent makes each line its own log. Tell it how a new event begins:

YAML
logs:
  - type: file
    path: /var/log/storefront/app.log
    service: storefront
    source: java
    log_processing_rules:
      - type: multi_line
        name: new_log_starts_with_date
        pattern: \d{4}-\d{2}-\d{2}

A line that does not match the pattern is appended to the previous event. Alternatively, log in JSON with one object per line, which avoids the problem completely and is the recommended format.

Filtering and masking at the source

log_processing_rules can also drop or redact data before it leaves the host:

YAML
    log_processing_rules:
      - type: exclude_at_match
        name: drop_healthchecks
        pattern: GET /healthz
      - type: mask_sequences
        name: mask_card_numbers
        replace_placeholder: "[masked_card]"
        pattern: \b(?:\d[ -]*?){13,16}\b
  • exclude_at_match drops matching lines. include_at_match keeps only matching lines.
  • mask_sequences replaces matches with a placeholder.

Filtering at the source saves ingestion cost and keeps sensitive data from leaving. Mask or avoid logging secrets in the first place.

Good log hygiene

  1. Log JSON with fields like level, message, trace_id, user_id.
  2. Include a trace ID so logs link to traces.
  3. Set service, env and version consistently.
  4. Never log passwords, tokens or full card numbers.
  5. Do not log at debug level in production by default.
Python Stand-in: multi-line grouping by a start pattern

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: exclude_at_match and mask_sequences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Troubleshooting checklist

  1. logs_enabled: true set and Agent restarted?
  2. datadog-agent status shows the source as OK?
  3. dd-agent can read the file (sudo -u dd-agent head /var/log/...)?
  4. Outbound access to the logs intake for your site (TCP 443)?
  5. Logs arrive but unparsed: wrong source, or no pipeline (next lesson).
  6. Logs missing from search: an exclusion filter or index quota (see the last logs lesson).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What must be true before the Agent can tail logs at all?

    Log collection is disabled by default and is enabled with logs_enabled.
  2. 2. What does a recognized source value such as nginx trigger?

    The source tells Datadog which integration pipeline to apply.
  3. 3. How are multi-line stack traces kept together?

    A multi_line rule defines how a new event starts; other lines append to the previous event.
  4. 4. Where should containerized apps write logs?

    The runtime captures stdout and stderr and the Agent collects them.

Summary

  • Enable logs_enabled, then configure sources: files, TCP/UDP, journald, Windows events, containers, HTTP and cloud forwarders.
  • service and source drive tagging and parsing; ensure dd-agent can read the files.
  • Use multi_line, exclude_at_match and mask_sequences to shape and protect data at the source.
  • Prefer JSON logs with a trace ID, and never log secrets.

Next: Log Pipelines and Parsing