Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 40 ยท Scaling and Operating Jenkins

Backup, Monitoring and Troubleshooting

Back up JENKINS_HOME, watch Jenkins with Prometheus and logs, and fix the usual trouble: full disks, stuck queues, hung builds, out-of-memory errors and slow UIs.

Once a team depends on Jenkins, it becomes production infrastructure: when it is down, nobody can ship. Operating it well comes down to three habits. Back up what matters and test the restore. Watch a few numbers so you hear about trouble before your users do. And know the short list of failures that account for most incidents, with a clear order in which to check them.

This lesson covers all three. The real server cannot run on this site, so commands and configuration are read-only examples, and the runnable parts are Node stand-ins that model the diagnosis.

The operator's routine

Back up Config, secrets and job definitions.
Watch Queue, executors, disk, heap.
Diagnose Read the reason Jenkins prints.
Fix Smallest change that works.

Most outages are disk, memory or a stuck queue.

1 / 2

What to back up

Everything the controller knows lives under JENKINS_HOME (see Jenkins Architecture). You rarely need all of it. The valuable, hard-to-recreate parts are:

  • config.xml and the other *.xml files in the top folder (global settings, plugin settings),
  • credentials.xml and the secrets/ folder. The credentials file is encrypted with keys from secrets/. One without the other is useless,
  • jobs/*/config.xml (job definitions) and nodes/ (agents),
  • users/ and, if you track it, the list of installed plugins and versions.

What you can usually skip: workspace/ (rebuilt by the next build), caches and war/, and, depending on your policy, old build logs and artifacts under jobs/*/builds/. Build history is nice to have but is the largest part.

If you use Configuration as Code, the controller settings are already in Git, and the backup mainly protects secrets, job history and anything JCasC does not cover.

Practical options, from simple to robust:

  • A scheduled copy with rsync or tar, excluding the bulky folders.
  • The ThinBackup plugin, which writes timestamped copies of the configuration on a schedule.
  • A volume snapshot of the disk or Docker volume (cloud snapshots, LVM, ZFS). This is fast and consistent enough for most teams.

A read-only example of a nightly script:

Terminal
#!/usr/bin/env bash
set -euo pipefail

SRC=/var/lib/jenkins
DEST=/backups/jenkins/$(date +%F)

mkdir -p "$DEST"
tar --exclude='workspace' --exclude='caches' --exclude='war' \
    --exclude='jobs/*/builds' \
    -czf "$DEST/jenkins-home.tar.gz" -C "$SRC" .

# keep two weeks
find /backups/jenkins -maxdepth 1 -type d -mtime +14 -exec rm -rf {} +

Two rules matter more than the tool. Store backups off the server (a different disk or machine), and practice a restore on a scratch machine. An untested backup is a hope, not a backup. Also protect the backup like a password vault, because it contains secrets/.

Monitoring with metrics

Jenkins can expose numbers for a monitoring system. The Prometheus metrics plugin publishes a text endpoint (by default at /prometheus/ on your Jenkins URL) that a Prometheus server scrapes on a schedule, and Grafana can chart it. The Metrics and Monitoring plugins offer similar data and a built-in page.

Useful signals to watch and alert on:

Signal Why it matters
Build queue length and time in queue Growing queue means too few executors or an offline agent
Executors busy vs. total Near 100% for long periods means you need capacity
Agents offline A label with no online agent blocks every job that needs it
Disk free on the controller and agents Full disk is the classic cause of a sudden outage
JVM heap used A line that only climbs hints at a leak or an undersized heap
Build duration and failure rate Shows slow regressions and flaky jobs

Exact metric names depend on the plugin and its version, so open the endpoint in a browser and look at what your installation exposes before writing alert rules. A scrape configuration looks like this (read-only):

YAML
scrape_configs:
  - job_name: jenkins
    metrics_path: /prometheus/
    scheme: https
    static_configs:
      - targets: ["jenkins.example.com"]

Alerts should be few and actionable: "disk under 15% free", "queue older than 30 minutes", "no executors online for a label".

Logs

Jenkins has three kinds of logs, and knowing which one answers your question saves time:

  • Build console output for one build, on the build page. It answers "why did my build fail".
  • The Jenkins system log, at Manage Jenkins > System Log, shows warnings from the controller itself, such as plugin loading errors, agent connection failures and SCM polling problems. You can add a logger for a specific class at a lower level (for example FINE) to learn more, then remove it.
  • The process log from the service manager: journalctl -u jenkins for a Linux package install, or docker logs <container> for a container. This is the only place to see startup failures when the web UI will not even load.

Troubleshooting the usual suspects

The disk is full

Symptoms: builds fail with No space left on device, Jenkins refuses to save configuration, or the UI shows errors. Find what is big and fix the cause, not just the symptom:

Terminal
df -h /var/lib/jenkins
du -sh /var/lib/jenkins/* | sort -h | tail
du -sh /var/lib/jenkins/jobs/* | sort -h | tail

Typical causes are build logs and artifacts that are never discarded (set a build discarder, see Artifacts, Stash and Fingerprints), workspaces that accumulate (clean them with cleanWs() or a workspace-cleanup policy), and Docker images piling up on agents (prune them on a schedule).

The queue is stuck

An item waits in the queue and never starts. Hover over it or open the queue page; Jenkins prints the reason. The common ones:

  • Waiting for next available executor means all matching executors are busy. Add capacity or reduce parallelism.
  • There are no nodes with the label 'x' means a typo in the label, or the agents with that label are offline. Check the agent list under Manage Jenkins.
  • Waiting for next available executor on (agent) with that agent offline: read the offline reason on the node's page (disk space, SSH failure, expired credentials).
  • Quiet period is normal; the job is waiting a few seconds so that several triggers can merge into one build.
  • Jenkins is about to shut down: the controller is in "quiet down" mode (someone chose "Prepare for shutdown") and starts nothing new until it is cancelled.
  • A lock or a throttle setting on the job is holding it back on purpose.

A build hangs

A build that has run for hours usually waits for something: a command that prompts for input, a network call with no timeout, or an input step nobody answered. Prevention is easy: put an overall timeout on every pipeline (see Declarative Pipeline Syntax and Common Steps), and use timeouts and non-interactive flags on commands:

GROOVY
options {
  timeout(time: 30, unit: 'MINUTES')
}

To investigate a live hang, look at the last console lines, open the Pipeline Steps view to see which step is running, and if needed run jstack against the agent process or inspect the process on the agent machine. As a last resort, abort the build; an abort that does not take effect can often be forced from the build's URL with /stop, /term or /kill.

Out of memory and a slow UI

Messages like java.lang.OutOfMemoryError: Java heap space or a UI that takes many seconds per page usually mean the controller's JVM is too small or overloaded. Check the heap graph, then set a larger limit through the service environment (for example JAVA_OPTS="-Xmx4g" in a Docker environment, or the systemd override of the package) and restart. Heap is a stopgap if the real cause is load, so also look for:

  • thousands of retained builds per job (set discarders),
  • many jobs polling SCM every minute (use webhooks, see Triggers),
  • builds running on the controller itself,
  • plugins with known memory problems (update them, or remove abandoned ones, see Plugins).

Other messages you will meet

  • Could not find credentials entry with ID 'x': the ID is misspelled or the credential's scope does not include this job (see Credentials and Secrets).
  • Offline: Disk space is too low on a node: Jenkins takes the node out of rotation to protect it. Free space and bring it back.
  • Resuming build at ... after Jenkins restart: normal; pipelines survive a controller restart, though a step that was running on an agent may be interrupted.

A stand-in queue doctor

The first stand-in turns the advice above into code. It receives a queue entry plus the state of the agents and prints the likely cause and next step. The rules are simplified, but the order of questions is how an experienced operator thinks: is the label known, is anyone online with it, is there a free executor.

JS A stand-in that explains why a build is waiting

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Look at the first line: linux has two agents, one is offline, and the other has both executors busy, so the job waits for a free slot. A different message would need a different fix, which is the whole point of reading the reason.

A stand-in disk and heap watcher

The second stand-in evaluates sampled metrics against alert rules, the way a monitoring system does. It also shows a simple trend check: a heap that grows at every sample after each garbage collection is a leak suspect.

JS A stand-in alert evaluator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Only the last sample fires, because both thresholds are crossed at minute 30. A good alert gives you time to act, which is why teams often add an earlier warning at 25% disk free.

Never "fix" a full disk by deleting files inside JENKINS_HOME at random. Removing the wrong file in secrets/ or a job's config.xml can make credentials or jobs unrecoverable. Delete old build folders and workspaces, or use the discard settings.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which part of JENKINS_HOME must be backed up together with credentials.xml?

    credentials.xml is encrypted with keys stored in secrets/. Without those keys the credentials cannot be decrypted after a restore.
  2. 2. A build has been running for 9 hours waiting on a command that prompts for input. What prevents this next time?

    A timeout aborts builds that stop making progress, and non-interactive flags stop commands from waiting for a person.
  3. 3. A queued job shows 'There are no nodes with the label gpu'. What is the most likely cause?

    That message is printed when no node carries the requested label at all. Fix the label or add the agent.
  4. 4. What is the best test of a Jenkins backup?

    Only a real restore proves the backup is complete, including the secrets that make credentials readable.
  5. 5. Where do you find startup errors when the Jenkins web UI will not load?

    If the UI is down, the System Log page is unavailable too. The service manager's log still has the startup output.

Summary

  • Back up config.xml, credentials.xml with secrets/, job and node definitions; skip workspaces and caches; store copies off-server and test restores.
  • Watch queue age, executor use, offline agents, disk space and JVM heap; the Prometheus plugin exposes them at /prometheus/.
  • Use the console output for a build, the System Log for the controller, and the process log for startup failures.
  • Read the queue reason first: it names missing labels, offline agents, busy executors, quiet period or shutdown mode.
  • Prevent hung builds with timeouts, full disks with discarders and cleanup, and memory trouble by sizing the heap and reducing load.

Next, turn a tested pipeline into safe releases in Deployment Pipelines and Approvals.