Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Scale and Operations

Troubleshooting Chef Runs and Why-Run

A practical guide to debugging Chef runs: reading errors, log levels, why-run mode, chef-shell, common failure patterns, and a runnable why-run model in Ruby.

Sooner or later a Chef run fails, or worse, succeeds and does something unexpected. Good troubleshooting is a method, not a bag of tricks: read the error carefully, find which phase failed, reproduce it on one machine, and look at the facts (attributes, resource properties, logs) before changing code.

This lesson gives you that method and the tools: reading a run's output, log levels, why-run mode, chef-shell, and a catalog of common failures and their fixes. The runnable example models why-run in Ruby. The commands themselves are real Chef and read-only here.

Read the error from the bottom

A failed run prints a block like this at the end:

TEXT
================================================================================
Error executing action `install` on resource 'package[nginx]'
================================================================================

Chef::Exceptions::Package
-------------------------
No candidate version available for nginx

Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/myapp/recipes/default.rb

  8: package 'nginx' do
  9:   action :install
 10: end

Four facts matter: which resource (package[nginx]), which action (install), the exception (No candidate version available) and where it was declared (file and line). The message here says the package manager cannot find nginx: the apt cache is stale (apt_update is missing) or the repository is not configured. Fix the cause, not the symptom.

Also determine the phase:

  • Errors before any resource runs, with a Ruby stack trace in recipes/... or attributes/..., happen during compile: syntax errors, nil values (undefined method '[]' for nil means a missing attribute), missing cookbooks.
  • Errors with "Error executing action" happen during converge: the system refused something.
  • Errors about the server (401, 403, 404, 412, 500) are API problems: wrong key, wrong organization, missing cookbook version or role.

Log levels and output

Terminal
chef-client -l debug              # very detailed (debug), noisy but honest
chef-client -l info -L /tmp/chef.log
chef-client --why-run             # show what would change
chef-client --local-mode --override-runlist 'recipe[myapp::default]'
chef-client -o 'recipe[myapp::default]' -z   # shorthand: local mode, run only this recipe

Levels: fatal, error, warn, info, debug. Use debug to see search queries, attribute reads and HTTP calls. In recipes, log with Chef::Log.info("port is #{node['myapp']['port']}") or the log resource.

Why-run

Why-run is a dry run. Chef walks the resource collection, compares the desired state with the machine, and prints what it would do, without changing anything:

TEXT
Recipe: myapp::default
  * package[nginx] action install
    - Would install version 1.24.0 of package nginx
  * template[/etc/myapp/app.conf] action create
    - Would update content in file /etc/myapp/app.conf
      --- /etc/myapp/app.conf       2025-02-03 09:12:01
      +++ /tmp/chef-rendered...
      @@ -1,2 +1,2 @@
      -port=8080
      +port=9090

It is not perfect: a resource may depend on a change an earlier resource would have made, so later resources may report errors or be skipped. Still, it is an excellent first look before a risky change.

Ruby Stand-in: a why-run mode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The why-run pass prints "Would install" and a small diff, then confirms that the machine is untouched. The real run applies the changes, and a later why-run reports everything up to date. That last step is a quick idempotence check you can also do on a real node.

chef-shell: look inside a run

chef-shell is an interactive Ruby session with the Chef libraries loaded. Run it on the node:

Terminal
sudo chef-shell -z              # local mode session
chef > node['platform']
 => "ubuntu"
chef > recipe_mode              # type resources and run them one by one
chef:recipe > package 'nginx'
chef:recipe > run_chef          # converge what you typed
chef > node.debug_value('myapp', 'port')

It is ideal for questions like "what does node['network']['interfaces'] really look like?" or "does this guard return true?" The breakpoint resource stops a run inside chef-shell for inspection (chef-client -s loads it).

A catalog of common failures

Symptom Likely cause Fix
undefined method '[]' for nil Missing attribute or typo in a nested key Check the key path; use node.dig('a', 'b'); set a default
401 Unauthorized Client key does not match the server Re-bootstrap or re-register the node key
403 Forbidden Permissions on the organization Add the client to the right group
412 Precondition Failed Cookbook version constraint unsatisfiable Compare environment, depends and uploaded versions
Cookbook not found Not uploaded or not in the run list knife cookbook list; upload; check metadata.rb name
Package not found Stale package cache or wrong repo Run apt_update first; verify repo resources
A resource runs every time Not idempotent: no guard on execute Add not_if/creates, or use a real resource
Service will not start Config error Check journalctl, verify the config in the template
Template shows old values Attribute precedence knife node show -l, debug_value
Run is very slow Heavy search, many Ohai plugins, large downloads Debug log with timing; filter_result; disable plugins
Delayed notification did not fire The run failed before the end Fix the failing resource first

A method that works

  1. Reproduce on one node, in Test Kitchen if possible: kitchen converge with -l debug.
  2. Narrow down with chef-client -o 'recipe[x::y]' -z to run only the suspect recipe.
  3. Print the facts: attributes, search results, resource properties.
  4. Change one thing at a time, and re-run with why-run first.
  5. Write a test (ChefSpec or InSpec) so the bug cannot return.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does chef-client --why-run do?

    Why-run compares desired and actual state and prints the intended actions without applying them.
  2. 2. An error mentions undefined method [] for nil in a recipe. What is the likely cause?

    Indexing into nil means a parent key is missing. Check attribute names and defaults.
  3. 3. Which tool lets you inspect node data interactively in a Ruby shell?

    chef-shell loads the Chef libraries so you can read the node object and run resources by hand.
  4. 4. Why should you reproduce a failure in Test Kitchen first?

    A throwaway instance lets you iterate quickly and safely.

Summary

  • Read the error for the resource, action, exception and file; identify whether it happened at compile, converge or API level.
  • Use -l debug, --why-run, -o ... -z and chef-shell to see facts and test ideas safely.
  • Common failures: nil attributes, stale package caches, version constraints, missing guards, permissions on the server.
  • Work methodically: reproduce, narrow, print, change one thing, add a test.

Next: Project: A Web Server Cookbook