Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 38 ยท Scale and Operations

Compile Servers and Scaling Puppet

Plan Puppet capacity: how agent runs load the server, JRuby instances and heap size, run interval and splay, compile servers with a load balancer, CA placement and caching.

A single Puppet server comfortably serves a few hundred nodes. Past that, you have to understand what the server actually does for each agent, because that tells you what to scale. The expensive part is catalog compilation: evaluating your manifests, Hiera data and templates for one specific node. Everything else (serving files, receiving reports) is comparatively cheap.

This lesson builds a capacity model, covers the main tuning knobs, and describes the architecture that large fleets use: a primary server with extra compile servers behind a load balancer. These are operations topics that cannot be run on this site, so the code blocks are read-only examples, and the runnable blocks are JavaScript stand-ins that calculate capacity.

What one agent run costs

Each run (by default every 30 minutes) consists of:

  1. The agent sends facts and requests a catalog. The server compiles it: CPU-heavy, typically seconds, and the one step that needs a JRuby instance.
  2. The agent downloads file content it needs (source files, plugin sync), mostly metadata and checksums.
  3. The agent applies the catalog locally. No server load.
  4. The agent posts a report, which the server stores or forwards to PuppetDB.

The server runs on the JVM. Puppet Server executes Ruby code in a pool of JRuby instances, and each instance compiles one catalog at a time. So the number of instances (max-active-instances) limits parallel compilations, and the heap size limits how many fit in memory.

A capacity model

Let N be the number of nodes, T the run interval (default 30 minutes) and c the average compile time. The average compile load is:

TEXT
compiles per minute = N / T(minutes)
busy instances      = compiles per minute x c(minutes)

For 3,000 nodes, a 30 minute interval and a 4 second compile: 100 compiles a minute, each taking 4/60 minute, so about 6.7 instances are busy on average. Real traffic is lumpy (after a deploy everyone gets new catalogs), so provision at least double.

The model also shows the cheapest optimizations: make compiles faster (fewer Hiera lookups per resource, avoid huge file trees and PuppetDB queries in loops) or spread the runs over time.

flowchart LR
  A["Agents"] --> LB["Load balancer"]
  LB --> C1["Compile server 1"]
  LB --> C2["Compile server 2"]
  LB --> C3["Compile server 3"]
  C1 --> P["Primary: CA, PuppetDB"]
  C2 --> P
  C3 --> P

Compile servers share the compilation load; the primary keeps the CA and the database.

Tuning one server

Settings live in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf and in the JVM arguments:

TEXT
# puppetserver.conf (jruby-puppet section)
max-active-instances: 6
max-requests-per-instance: 100000

# JVM heap, in /etc/default/puppetserver (or /etc/sysconfig/puppetserver)
JAVA_ARGS="-Xms4g -Xmx4g"

Guidelines:

  • Instances should not exceed the number of CPU cores. Beyond that, instances fight for CPU, and each compile gets slower.
  • Heap grows with instances: allow roughly 512 MB to 1 GB per instance plus overhead for the JVM and the code cache; set -Xms equal to -Xmx to avoid resizing.
  • Raise max-requests-per-instance only if you see no memory growth; a lower value recycles instances and hides leaks.

Spreading the load: interval and splay

If 3,000 agents all started their run at 10:00 sharp, the server would get all requests in the same minute. Agents have a splay setting that adds a random delay, and a runinterval:

INI
# puppet.conf on the agents
[agent]
runinterval = 30m
splay = true
splaylimit = 30m

Cron-driven or event-driven runs (for example a Bolt command that triggers an immediate puppet agent -t on selected nodes) are fine for urgent changes, but run them in controlled batches.

Compile servers and the CA

Add capacity horizontally with compile servers: additional Puppet Server hosts that run the same code and compile catalogs. Agents connect to a load-balancer name (for example puppet.example.com), and the load balancer spreads the requests round-robin.

What has to be shared or centralized:

Concern How
Code Each compile server runs r10k/Code Manager and deploys the same control repo commit
Certificate Authority One CA (the primary) signs all certificates; compile servers proxy certificate requests to it, or hold CA-signed certificates with the load balancer's name in their alt names
Reports and facts Sent to PuppetDB on the primary, or to a PuppetDB per site
Hiera data Part of the control repo; encrypted data needs the key on every compile server

Placement tips:

  • Put compile servers near the agents (per datacenter or region) to avoid slow links; a remote catalog compile is chatty.
  • Keep the CA and PuppetDB highly available according to your needs; they are the shared single points.
  • Treat a compile server as disposable: rebuilding one should take a few commands, not a day.

Reducing load without hardware

  • Cache catalogs for nodes that rarely change with puppet agent --use_cached_catalog in carefully chosen scenarios, or the server-side catalog_cache_terminus, at the cost of staleness.
  • Trim data lookups: thousands of Hiera lookups with deep hierarchies add up.
  • Use source sparingly: shipping big trees from the server is slower than installing a package.
  • Fix compile errors fast: failing compiles retry and waste capacity.
  • Profile. puppetserver exposes metrics (JRuby borrow time, compile time) and the agent's --evaltrace and --profile options reveal slow resources.

Stand-ins: capacity calculators

The first block implements the capacity model above, with a safety factor. The second simulates splay and shows how random delays flatten the peak. Both are JavaScript stand-ins that teach the arithmetic.

JS Stand-in: how many JRuby instances do I need?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last line shows why compile time matters: tripling it triples the fleet you need.

JS Stand-in: splay flattens the peak

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which part of an agent run puts the main load on the Puppet server?

    Compilation runs on the server in a JRuby instance; applying and collecting facts happen on the agent.
  2. 2. What should max-active-instances roughly not exceed?

    More instances than cores slows every compile because they compete for CPU.
  3. 3. Why enable splay on agents?

    A random delay flattens request peaks on the server.
  4. 4. What must stay centralized when you add compile servers?

    One CA signs all certificates; compile servers proxy to it or trust it.

Summary

  • Compilation is the expensive step; it runs in JRuby instances limited by cores and heap.
  • Capacity is roughly nodes / interval x compile time, multiplied by a safety factor.
  • Use splay and sensible run intervals to avoid spikes.
  • Compile servers behind a load balancer add capacity; the CA and PuppetDB remain shared.
  • Faster compiles (fewer lookups, smaller catalogs) are the cheapest scaling.

Next: PuppetDB and Reporting