Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Scale and Operations

Splay, Intervals and Scaling Chef Runs

Learn how chef-client interval and splay prevent a thundering herd, how load grows with node count, ways to scale the Chef Server, and simulate splay in Ruby.

With ten servers, nobody worries about when chef-client runs. With ten thousand, the question "who talks to the server at the same moment?" decides whether your Chef Server stays healthy. If every node runs on the hour, the server sees 10,000 requests in a few seconds and then nothing for the rest of the hour. This lesson explains how interval and splay spread the load, how to estimate capacity, and what to do when one server is not enough.

The Chef Server is not available on this site, so the configuration is read-only. A Ruby simulation lets you see the thundering herd and the effect of splay with real numbers.

Interval and splay

When chef-client runs as a daemon or service, two settings control the schedule:

Ruby
# /etc/chef/client.rb
interval 1800     # seconds between runs: every 30 minutes
splay 300         # add a random delay of 0 to 300 seconds before each run
log_level :warn

With interval 1800 and splay 300, each node waits 1800 seconds plus a random 0 to 300 seconds. The exact delay differs per node and per run, so nodes drift apart over time and the load flattens. Systemd timers, cron jobs and the chef_client cookbook (or the newer chef_client_systemd_timer resource) can do the same:

Ruby
chef_client_systemd_timer 'Run Chef Infra Client every 30 minutes' do
  interval '30min'
  delay_after_boot '5min'
  splay 600
end

Rules of thumb: choose an interval you can live with (30 to 60 minutes is common), and make splay a good fraction of it (a few minutes is fine for hundreds of nodes; use more when you have thousands).

Estimating load

Each run makes a series of API calls: authenticate, expand the run list, check cookbooks, update node data. Say one run keeps the server busy for about 1.5 seconds of capacity. Then:

  • Runs per hour = nodes x (3600 / interval).
  • Average concurrent runs = runs per second x seconds per run.

For 10,000 nodes at 30 minutes, that is 20,000 runs per hour, about 5.6 per second, so roughly 8 concurrent runs on average with a 1.5 second run cost. That is easy when evenly spread, and impossible if all of them arrive within the same ten seconds.

Ruby Simulation: thundering herd versus splay

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

With no splay all 3,000 nodes start in the same second, so the peak equals 3,000. With a splay of 30 seconds it drops to 225, with a splay of 300 to 35, and at the full interval to 11, against a perfectly even average of about 3. The details depend on the random seed, but the trend is what matters: more splay, flatter load.

Making each run cheaper

The cheapest request is the one you do not make. Ways to reduce server work per node:

  • Slow down. Is a 15-minute interval needed? Many fleets run hourly and trigger manual runs for urgent changes (with knife ssh or a push job).
  • Fewer cookbook downloads. Nodes cache cookbooks and only fetch changed files; avoid versions that change on every upload.
  • Avoid heavy searches in recipes. A search that returns all node objects for every node on every run is the usual hidden cost. Use filter_result.
  • Smaller node objects. Large attribute trees (huge Ohai plugins, saved debug data) slow saving. Disable unneeded Ohai plugins (ohai.optional_plugins, ohai.disabled_plugins) and use automatic_attribute_whitelist / allowlists so only needed attributes are saved.
  • Chef Infra Client local mode (chef-client -z with Policyfile archives) removes the server from the run entirely for some workloads.
  • Policyfiles avoid dependency solving on the server for each run, which was historically a major cost.

Scaling the server side

flowchart LR
  A["Standalone: one machine, up to a few hundred or thousand nodes"] --> B["Tiered: separate front end, back end and search"]
  B --> C["High availability: several front ends + a load balancer, PostgreSQL and search replicated"]
  C --> D["Chef Automate or Cinc + object storage, split by organization"]

Grow in steps. Measure before moving to the next one.

  • Standalone is fine for small fleets. Give it enough RAM and fast disk; the search index and PostgreSQL are the hungry parts.
  • Tiered / HA topologies put several front-end servers behind a load balancer and keep the database and search tier on their own machines.
  • Object storage (S3-compatible) can hold cookbook files instead of local disk.
  • Monitor: request latency, 5xx counts, PostgreSQL connections, search queue length and disk. Chef Server exposes status endpoints, and the data collector and Automate show run duration trends.
  • Split organizations if teams have very different needs, since each organization has its own cookbooks and nodes.

Chef Infra Server is one option. Chef Automate and Cinc (a free, community-built distribution of the same source) offer similar topologies. Some teams avoid a server entirely and run Chef in local mode from a Git checkout or a Policyfile archive delivered with a package or from object storage.

Operations checklist for a large fleet

  1. Set interval and splay in client.rb; verify with systemctl list-timers or the service config.
  2. Alert when nodes have not checked in for a few intervals (a stale ohai_time).
  3. Track runs that take much longer than normal.
  4. Pin versions with Policyfiles so a bad upload cannot hit all nodes at once.
  5. Roll out cookbook changes in waves: policy groups for canary nodes first.
  6. Keep an eye on node object size and search indexing delay.
Ruby Simulation: finding silent nodes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The alert rule is deliberately generous (three intervals) so a slow run or a short network issue does not page anyone, yet a node that is really gone is found within a couple of hours.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does splay do?

    Splay adds a random wait of 0 to the splay value, spreading the load over time.
  2. 2. With 6,000 nodes and a 30 minute interval, how many runs start per hour on average?

    Two runs per node per hour (3600 / 1800) times 6,000 nodes is 12,000.
  3. 3. Which change reduces server work for each run?

    Smaller searches and smaller node objects cut CPU, network and index work.
  4. 4. What is a sensible way to roll out a risky cookbook change in a big fleet?

    Waves limit the blast radius and give you time to notice problems.

Summary

  • interval sets how often chef-client runs; splay adds a random delay to flatten the load.
  • Estimate capacity as nodes times runs per hour times seconds per run; avoid synchronized runs.
  • Cut per-run cost with fewer searches, smaller node objects, Policyfiles and sensible intervals.
  • Scale the server in steps (standalone, tiered, high availability), monitor, and roll out changes in waves.

Next: Troubleshooting Chef and Why-Run