Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 40 ยท Scaling and Operating Jenkins

Agents, Labels and Scaling

Connect SSH and inbound agents, route builds with labels, use cloud and Kubernetes ephemeral agents, size executors and throttle jobs as your Jenkins grows.

One Jenkins controller cannot do all the work for long. Builds compete for CPU and disk, different projects need different tools (a Windows box, a macOS machine, a GPU server), and running builds on the controller is a security risk. The answer is agents: other machines or containers that run the actual work while the controller schedules and records it. Jenkins Architecture introduced agents and executors; this lesson shows how to connect them, how labels decide which agent a stage gets, and how large installations scale.

The real nodes and clouds are described in read-only snippets. The runnable stand-ins are a label expression matcher and a queue scheduler that places builds on agents with free executors.

From one box to a fleet

Controller Schedules, never builds.
Static agents Long-lived machines you manage.
Cloud agents Started on demand, then deleted.

Most teams end up with ephemeral agents.

1 / 2

Connecting an agent

An agent needs Java and a way to talk to the controller. There are three common connection methods.

SSH agents. The controller connects out to the agent over SSH, using an SSH credential, and starts the agent process itself. Setup is simple: create the node in Manage Jenkins > Nodes, give the host, credential and a remote directory (the workspace root). The agent machine needs Java and an SSH server, and the controller must be able to reach it.

Inbound (formerly "JNLP") agents. The agent connects to the controller. Use this when the agent is behind a firewall or NAT that the controller cannot reach. You create the node in Jenkins, then run the agent on the machine with the secret Jenkins shows you:

Terminal
java -jar agent.jar \
  -url https://jenkins.example.com/ \
  -name build-linux-1 \
  -secret <the-secret-from-the-node-page> \
  -workDir /home/jenkins/agent

Depending on your Jenkins version and network, the agent connects with the classic TCP port (50000 by default) or over WebSocket through the normal HTTPS port, which avoids opening an extra port. Run the process as a service so it restarts after reboots; the official jenkins/inbound-agent Docker image does this job for container setups.

Cloud-provisioned agents. A plugin (Amazon EC2, Azure VM, Google Compute, Kubernetes, Docker and more) creates agents when the queue needs them and destroys them when idle. You configure a cloud and one or more templates; Jenkins does the rest.

Whichever method you use, the agent runs with the permissions of its operating-system user. Give it only what the builds need, and keep sensitive secrets in Jenkins credentials rather than baked into the machine (Credentials and Secrets).

Labels and node selection

A label is a free-form tag on an agent: linux, windows, docker, gpu, jdk21, arm64. A pipeline asks for what it needs:

GROOVY
pipeline {
  agent none
  stages {
    stage('Build') {
      agent { label 'linux && docker' }
      steps { sh 'make' }
    }
    stage('Windows installer') {
      agent { label 'windows && !gpu' }
      steps { bat 'build.cmd' }
    }
  }
}

A label expression can use && (and), || (or), ! (not) and parentheses. Jenkins sends the stage to an agent whose labels satisfy the expression and that has an idle executor. If none matches, the build waits in the queue, with a message such as "There are no nodes with the label 'windows'" when the label does not exist at all.

Good label design makes pipelines portable: label by capability (docker, jdk21) instead of by machine name (build-linux-3), so you can swap or add machines without touching Jenkinsfiles. The stand-in below parses label expressions the way Jenkins reads them, in a simplified form.

JS Stand-in: matching label expressions

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Each expression picks out the agents you would expect, and macos matches nobody, which is exactly the situation in which a real build waits in the queue.

Executors: how many per agent?

An executor is one slot that runs one build at a time on a node. The number of executors is a setting per node, and it should follow the machine's real capacity: usually close to the number of CPU cores for CPU-bound builds, fewer for memory-hungry ones. Too many executors cause builds to fight over CPU and memory and become slow and flaky; too few leave the machine idle while jobs queue.

Notes on sizing:

  • Set the built-in controller node to 0 executors (Security, Users and Authorization).
  • One build can use many cores internally (Maven or Gradle with parallel workers), so one executor on a big machine may be better than eight.
  • Executors share disk, so watch free space. Every workspace uses space until cleaned.
  • Parallel stages use extra executors; a pipeline with 8 parallel branches asks for 8 slots at once.

Ephemeral agents

A static agent lives for months. Over time it collects stale files, mismatched tool versions and leftovers from earlier builds, which is the classic cause of "it only fails on that agent". An ephemeral agent is created for a build (or a small number of builds) and deleted afterwards. Every run starts from a known image, which gives reproducibility and security: nothing sensitive stays behind.

The Kubernetes plugin is the best-known example. It runs each agent as a pod, described by a pod template:

GROOVY
pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
            - name: maven
              image: maven:3.9-eclipse-temurin-21
              command: [sleep]
              args: [infinity]
              resources:
                requests: { cpu: "1", memory: 2Gi }
                limits:   { cpu: "2", memory: 4Gi }
      '''
    }
  }
  stages {
    stage('Build') {
      steps { container('maven') { sh 'mvn -B verify' } }
    }
  }
}

The pod's resource requests tell Kubernetes how much capacity to reserve, so the cluster, not Jenkins, handles bin-packing. If the cluster needs a new node, it can start one (with a cluster autoscaler) and the build waits only a short time. VM-based cloud plugins work similarly but boot slower, so they often keep a small idle pool.

Trade-offs: ephemeral agents have a cold start (image pull) and no warm cache. Mitigate with small images, a registry mirror, a shared cache volume, or build tools that cache to a remote store (Docker Agents and Building Images discusses image caches).

Scheduling and the queue

The scheduler below is a rough model of what the controller does: builds wait in a queue in order, each needs a label, and an agent runs a build when it has a matching free executor. Each "tick" is one time unit. It shows why a scarce label (one Windows agent) becomes a bottleneck while Linux agents sit free.

Node.js Stand-in: a label-aware build scheduler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

You can see the second installer build waiting for the single Windows executor while Linux work keeps flowing. In real Jenkins the fix is either another Windows agent or a cloud template that can start one on demand.

Throttling and fairness

When many pipelines compete for scarce resources, keep control with:

  • options { disableConcurrentBuilds() } so a job never runs twice at once (for example a deployment).
  • lock (Lockable Resources plugin) to allow only one pipeline at a time to use a shared resource, such as a staging environment or a license.
  • Throttle Concurrent Builds plugin to cap how many builds of a category run across all agents.
  • Priorities and separate labels so critical release builds do not wait behind a flood of pull-request builds.
  • Timeouts (options { timeout(time: 30, unit: 'MINUTES') }) so a hung build releases its executor.

What is real and what is the stand-in

SSH and inbound agents, labels, executors, the Kubernetes plugin and the throttling options are real features; the snippets show real syntax but need a real controller and nodes to run. The two TryIt blocks are simplified stand-ins. The real scheduler also considers node usage preferences, load balancing, offline nodes and build quiet periods.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A team's agent sits behind a firewall that blocks inbound connections from the controller. Which agent type fits?

    Inbound agents initiate the connection themselves, so only outbound access to Jenkins is needed.
  2. 2. What does the label expression linux && !gpu select?

    && means both conditions, and ! negates the label that follows.
  3. 3. Why are ephemeral agents popular?

    Because the agent is thrown away, you avoid drift, stale files and leftover secrets. The trade-off is a cold start.
  4. 4. A build is waiting and the queue says there are no nodes with label 'macos'. What is the likely cause?

    Label matching needs an agent that has the label (and a free executor). Without one the build waits.

Summary

  • Agents do the work; the controller schedules. Connect them with SSH, inbound connections or cloud/Kubernetes plugins.
  • Labels describe capabilities; pipelines request them with expressions such as linux && docker.
  • Size executors to the machine's real capacity and keep the controller at zero.
  • Ephemeral agents give clean, reproducible builds at the cost of cold starts.
  • Use concurrency options, locks, throttling and timeouts to share scarce resources fairly.

Next, stop copy-pasting pipeline code between repositories with Shared Libraries.