Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 38 ยท Scale and Operations

PuppetDB and Reporting

Use PuppetDB to store facts, catalogs and reports: PQL queries, the query() function, exported resources, report processors, node status and dashboards for your fleet.

Every agent run produces a lot of information: the facts of the node, the catalog that was compiled and a report describing what happened when it was applied. If this data disappears after each run, you cannot answer simple fleet-wide questions such as "which nodes run Ubuntu 20.04?" or "which nodes failed the last run?". PuppetDB is the database that keeps this data and makes it queryable.

It stores facts, catalogs, reports and (optionally) exported resources, in PostgreSQL, behind a REST API. Puppet Server writes to it; you and your tools read from it. This lesson covers the query language, the way to use PuppetDB from manifests, and report handling. PuppetDB is a server component, so the examples are read-only, and the runnable blocks are JavaScript stand-ins that model queries over fact data.

What PuppetDB stores

Data Used for
Facts Inventory: find nodes by OS, memory, any fact
Catalogs See which resources and relationships each node has
Reports Run outcomes, changed resources, failures, timings
Resources and edges Search for a specific resource across the fleet
Events Individual changes ("file X changed from A to B")
Exported resources Resources one node publishes for others

Data is retained for a configurable time (a few weeks of reports is common); nodes that have not checked in for a long time are deactivated and eventually purged.

Querying: PQL and the AST

PuppetDB's native query is a JSON structure (the abstract syntax tree), but the friendlier Puppet Query Language (PQL) compiles to it. You use it with the puppet query command (from the puppet-query tooling) or via the API:

Terminal
puppet query 'nodes { facts_environment = "production" }'
puppet query 'inventory[certname] { facts.os.family = "Debian" and facts.memory.system.total_bytes < 4294967296 }'
puppet query 'resources[certname, title] { type = "Package" and title = "openssl" }'
puppet query 'reports[certname, status, end_time] { latest_report? = true and status = "failed" }'

PQL queries have the shape entity[fields] { filters }. The entities you use most: nodes, inventory, facts, resources, reports, events, catalogs. The second query reads: "show the nodes that are Debian family and have less than 4 GiB of memory." Questions like these used to need a separate configuration database.

Using PuppetDB inside manifests

From a manifest you can query PuppetDB with the puppetdb_query() function or the query_nodes() and query_facts() helpers from the puppetdb module:

PUPPET
$web_nodes = puppetdb_query('inventory[certname] { facts.networking.hostname ~ "^web" }')
$addresses = $web_nodes.map |$n| { $n['certname'] }

file { '/etc/haproxy/backends.cfg':
  ensure  => file,
  content => epp('profile/backends.cfg.epp', { 'backends' => $addresses }),
}

This is service discovery with no extra tools: a load balancer's configuration is generated from the list of web nodes. The price: the compile now depends on PuppetDB being up, and the data is only as fresh as the last run of those nodes.

Exported resources

A node can export a resource declaration and other nodes can collect it. The classic example: every web server announces itself, and the monitoring server collects them all into its configuration.

PUPPET
# on each web server
@@nagios_host { $facts['networking']['fqdn']:
  ensure  => present,
  address => $facts['networking']['ip'],
}

# on the monitoring server
Nagios_host <<| |>>

@@ exports, <<| |>> collects (the content between the bars is a filter on attributes). After a web server's first run the resource is in PuppetDB; the monitoring server picks it up on its next run. Hence eventual consistency: allow two run cycles. Exported resources are elegant but make compiles heavier and debugging harder. Many teams now prefer explicit puppetdb_query() data and normal templates.

Reports

At the end of a run the agent sends a report: run status (changed, unchanged, failed), timings, a list of resource events and logs. The server hands it to the configured report processors in puppet.conf:

INI
[server]
reports = puppetdb,log

puppetdb stores it; log writes it to the server log; community processors forward to Slack, Splunk or a ticket system. A fleet-wide view of reports answers: what failed, what changed unexpectedly, which nodes have not reported (stale or broken agents).

Useful report-driven alerts:

  • Failed run for a node two times in a row.
  • A node with no report for more than 2x the run interval.
  • Corrective changes (resources fixed after drift) above a threshold: someone is changing machines by hand.
  • Noop runs that report pending changes: input for a release decision.
flowchart LR
  A["Agent run"] -->|facts, report| S["Puppet Server"]
  S -->|stores| D[("PuppetDB")]
  D --> Q["PQL queries / API"]
  D --> X["Exported resources"]
  Q --> G["Dashboards and alerts"]

PuppetDB turns single runs into fleet-wide data.

Stand-ins: queries over data

The first stand-in implements a tiny query over an inventory, with a filter like the PQL example above. The second computes the report-based alerts. Both are JavaScript stand-ins: PuppetDB uses PostgreSQL and a real query planner.

JS Stand-in: inventory query like PQL

Press Run (or Ctrl + Enter) to see the console output here.

No output.

JS Stand-in: find failed and silent nodes from reports

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Operating PuppetDB

  • Back up its PostgreSQL database if you depend on its history; the data can otherwise be rebuilt from the next agent runs.
  • Watch the command queue (reports and catalogs waiting to be stored); a growing queue means the database is too slow.
  • Tune report retention (report-ttl) and node purge settings to keep the database small.
  • Restrict access: facts contain host names, addresses and sometimes secrets leaked through custom facts.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does PuppetDB store?

    PuppetDB keeps node data from agent runs and serves it through an API.
  2. 2. What does @@nagios_host { ... } do?

    @@ exports a resource to PuppetDB; <<| |>> collects it elsewhere.
  3. 3. Why is a freshly exported resource not visible to collectors immediately?

    Exported resources are eventually consistent: allow a full run cycle or two.
  4. 4. Which report-based alert catches an agent that quietly stopped?

    A node with no recent report is silent, which signals a broken agent or machine.

Summary

  • PuppetDB stores facts, catalogs, reports and events and exposes them through PQL and a REST API.
  • puppetdb_query() powers service discovery inside manifests, at the cost of a compile-time dependency.
  • Exported resources (@@ / <<| |>>) share resources between nodes with eventual consistency.
  • Report processors forward run results; alert on failures, silence and unexpected corrective changes.
  • Watch the command queue and retention to keep the database healthy.

Next: Orchestration with Bolt: Tasks and Plans