Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 38 ยท Scale and Operations

Troubleshooting and Debugging Puppet

Find and fix Puppet problems fast: reading agent output, --debug and --trace, certificate and SSL errors, compile errors, dependency cycles, duplicate declarations and slow runs.

Sooner or later a Puppet run fails, hangs or does something surprising. The people who stay calm in those moments have a method: decide which phase failed, read the message from the right place, reproduce it on one node, and change one thing at a time. This lesson gives you that method and a catalog of the errors that cause most incidents.

Remember the phases of a run: (1) the agent collects facts, (2) it contacts the server over SSL, (3) the server compiles the catalog, (4) the agent applies it, (5) a report is sent. Errors in each phase look different and are fixed in different places. These are command-line and server tasks, so the commands here are read-only, and the runnable blocks are JavaScript stand-ins that classify messages and detect dependency cycles.

Find the phase first

Symptom Phase Where to look
Could not retrieve facts or a Ruby error in a fact Facts Run facter on the node; test the custom fact alone
SSL_connect, certificate verify failed, Connection refused Connection / certificates Agent puppet.conf, DNS, clock, CA
Evaluation Error, Could not find class, Duplicate declaration Compile Server log, puppet lookup, the code
Error: /Stage[main]/.../Service[x]: Failed to apply Apply Agent output, the service or package logs
Run takes ages Any --debug, --evaltrace, server metrics

The agent prints the phase in the message. Compile errors begin with Error: Could not retrieve catalog from remote server; apply errors name the resource path with Stage[main].

The debugging tools

Terminal
puppet agent --test                       # one run in the foreground with output
puppet agent --test --noop                # show what would change, change nothing
puppet agent --test --debug               # very detailed
puppet agent --test --trace               # Ruby stack traces for errors
puppet agent --test --evaltrace           # time and order for every resource
puppet agent --test --environment feature_x
puppet apply -e 'notice($facts["os"]["family"])'
puppet lookup some::key --node web1.example.com --explain
puppet config print --section agent       # what settings is the agent really using?
puppet resource service nginx             # inspect the real state of one resource

--test is shorthand for several useful options (foreground, verbose, detailed exit codes, no daemon). Its detailed exit code helps scripts: 0 means no changes, 2 means changes applied, 4 means failures, 6 means both.

In your own code, print values while developing:

PUPPET
notice("port is ${port}")
notice(inline_epp('<%= $facts %>', {}))   # or: notice($facts['os'])

Remove such lines before merging, or use debug() which only prints with --debug.

Certificate and SSL problems

The most common first-day failure is SSL. Typical causes:

  • Name mismatch. The agent asks puppet but the server certificate does not list that name. Fix server = in puppet.conf or add the name to the server's dns_alt_names.
  • Clock skew. A certificate that is "not yet valid" because the node's clock is wrong. Fix NTP first.
  • Unsigned request. The agent's request is waiting for puppetserver ca sign --certname web1.example.com (or autosign policy).
  • Stale certificate after a rebuild. Clean the old one with puppetserver ca clean --certname web1.example.com and remove the agent's ssldir.
Terminal
puppetserver ca list              # pending requests
puppetserver ca sign --certname web1.example.com
puppetserver ca clean --certname web1.example.com

Compile errors

Could not find class / Unknown resource type. The module is not in the environment, or the Puppetfile is missing the dependency; deploy again and check the name.

Duplicate declaration. Two places declare Package['git']. Fix by declaring it once (for example in profile::base) or by using ensure_packages(['git']) from stdlib, which tolerates repeats.

Evaluation Error: Unknown variable. A fact or Hiera value is missing, often because the key has a typo. With strict_variables = true (recommended) this fails loudly instead of silently becoming empty.

Type mismatch on a parameter. Expected Integer, got String: the Hiera value is quoted or wrong. Read the full message: it names the class and parameter.

Dependency cycle. Found 1 dependency cycle: (File[a] => Service[b] => File[a]). Puppet can also write a graph with --graph. Break the loop by removing one relationship or splitting the resources.

Apply errors

Most are ordinary admin problems: a package not found in the repository, a service unit with the wrong name, a file path whose parent directory does not exist. Reproduce the failing step by hand with the same user, then fix the resource (require => File['/opt/app'], a better provider, or the data).

A resource that changes on every run deserves attention: an exec without creates, onlyif or unless; a file whose content includes a timestamp; a notify-chain loop. Idempotence matters, see Acceptance Testing.

Slow runs

  1. Compare phase timings in the agent's final "Notice: Applied catalog in X seconds" line and the server's compile time.
  2. Use --evaltrace to find the slowest resource, often a package install or an exec.
  3. On the server, large Hiera hierarchies and puppetdb_query() calls in loops dominate compile time, see Compile Servers and Scaling.

Stand-ins

The first stand-in classifies an error message into the phases above, a useful way to practise triage. The second detects a dependency cycle in a resource graph and prints it the way Puppet does. Both are JavaScript stand-ins, small enough to read in one go.

JS Stand-in: classify an error message by phase

Press Run (or Ctrl + Enter) to see the console output here.

No output.

JS Stand-in: find a dependency cycle

Press Run (or Ctrl + Enter) to see the console output here.

No output.

flowchart TD
  S["Run failed"] --> Q1{"Message mentions SSL or connection?"}
  Q1 -->|yes| C["Certificates, DNS, clock"]
  Q1 -->|no| Q2{"Compile error from server?"}
  Q2 -->|yes| CP["Code, data, modules: lint, lookup, validate"]
  Q2 -->|no| AP["Apply error: reproduce on the node by hand"]

Decide the phase first; the right tool follows.

A habit that pays off

Write down every incident's cause and fix in a short runbook entry. After six months you will have a searchable list that answers most pages at 3 a.m. And when a fix is "we should have had a test for this", add the unit test the same day.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A run fails with 'certificate verify failed'. Which is a likely cause?

    SSL errors arise from name mismatches, clock skew, unsigned or stale certificates.
  2. 2. Which flag prints the time and order of every resource during apply?

    --evaltrace helps find the slowest resources.
  3. 3. Two profiles both declare Package['git']. Best fix?

    Duplicate declarations are resolved by defining the resource in one place or using a tolerant helper.
  4. 4. What does exit code 2 from puppet agent --test --detailed-exitcodes mean?

    0 means no changes, 2 means changes applied, 4 failures, 6 both.

Summary

  • Identify the phase (facts, connection, compile, apply, report) before anything else.
  • Use puppet agent --test with --noop, --debug, --trace and --evaltrace.
  • SSL problems are mostly names, clocks and certificate state.
  • Compile errors point to missing classes, duplicates, bad data types or cycles.
  • Slow runs: measure first, then fix the heaviest step.

Next: Project: A Complete Web Server Stack