Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 29 ยท Real-World Practice

Performance, Dynamic Inventory and AWX

Make Ansible faster and scale it to big fleets: forks, pipelining, strategies, async tasks, fact caching, dynamic inventory plugins, ansible-pull and AWX.

Ansible is fast enough for ten servers with no tuning at all. At a few hundred servers, or with a playbook that takes twenty minutes, the defaults start to hurt. The good news is that most of the cost is easy to explain: Ansible opens SSH connections, copies small Python programs, runs them and reads the results back. Tuning means doing less of that, or doing more of it at the same time.

This lesson covers the settings that matter most (forks, pipelining, strategies, fact handling, async tasks), then the two things that change how you run Ansible at scale: dynamic inventory, where the host list comes from a cloud API instead of a file, and AWX / Ansible Automation Platform, a web service that runs your playbooks for a whole team. You should already know inventory and how a playbook runs.

Real Ansible needs servers and SSH, so every command, config file and playbook here is read-only. The runnable examples are JavaScript stand-ins: a timing model for forks and strategies, and a tiny inventory-plugin imitation. They copy the idea, not Ansible's code.

Where the time goes

A normal task on one host does roughly this: build a temporary module file, copy it to the managed node over SSH, run it with Python, collect the JSON result, delete the temporary file. Ansible does that for every task on every host. Two numbers decide the total: how many hosts run at the same time, and how many round trips each task needs.

flowchart LR
  P["Playbook<br/>(tasks 1..N)"] --> W["Worker pool<br/>(forks = 5)"]
  W --> H1["web1"]
  W --> H2["web2"]
  W --> H3["web3"]
  W --> H4["web4"]
  W --> H5["web5"]
  H5 --> Q["web6 ... web12<br/>wait for a free worker"]

Only forks hosts are worked on at any moment; the rest of the inventory queues behind them.

Forks: how many hosts at once

forks is the size of the worker pool, which means the number of hosts Ansible talks to in parallel. The default is 5, which is deliberately cautious. Raise it in ansible.cfg or per run:

INI
# ansible.cfg
[defaults]
forks = 20
Terminal
ansible-playbook site.yml -f 50

More forks use more memory and CPU on the control node (each fork is a process) and open more simultaneous SSH connections. A sensible approach is to raise it step by step while watching the control node, and to stop when the target machines, a bastion host or a firewall rate limit becomes the bottleneck.

Fewer round trips: pipelining and SSH multiplexing

SSH multiplexing is on by default: Ansible asks SSH to reuse one connection for many tasks (ControlMaster=auto, ControlPersist=60s), so it does not repeat the handshake for each task.

Pipelining removes the file copy. Instead of uploading the module to a temporary file, Ansible streams it through the SSH connection to the Python interpreter. That saves one or two round trips per task, which adds up.

INI
[ssh_connection]
pipelining = True

Pipelining has one classic catch: it fails if the managed node's sudoers has requiretty set, because there is no terminal to give the command. Modern distributions do not enable it by default, but old enterprise images sometimes do. If tasks that use become start failing after you enable pipelining, check that setting first.

Do less: facts and the setup step

Every play starts with "Gathering Facts", which runs the setup module on each host. That is often the slowest single step. You have four ways to reduce it:

YAML
- hosts: web
  gather_facts: false          # skip it completely if the play does not use facts
  tasks: []

- hosts: web
  gather_facts: true
  gather_subset:               # collect only what you need
    - "!all"
    - "!min"
    - network
  tasks: []

You can also cache facts so that a later run reuses them. With the jsonfile cache plugin, facts are stored as files on the control node:

INI
[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /var/tmp/ansible_facts
fact_caching_timeout = 3600

With gathering = smart, a host whose cached facts are still fresh is not asked again. The price is that the facts may be an hour old, so do not rely on cached values for things that change quickly (free memory, current IP leases). Redis and other cache plugins exist for shared caches.

Strategies: linear, free and friends

A strategy decides how tasks move through the host list. The default, linear, runs task 1 on all hosts (in groups of forks), waits until every host has finished it, then starts task 2. free lets each host run ahead on its own, so a fast host does not wait for a slow one.

YAML
- hosts: web
  strategy: free
  tasks:
    - name: Install packages
      ansible.builtin.package:
        name: nginx
        state: present

free finishes sooner when hosts differ a lot in speed. It makes the output harder to read, because lines from different hosts interleave, and it is wrong whenever one host's progress should wait for the others (for example, take a node out of a load balancer on every host before upgrading any of them). There is also host_pinned, which keeps a worker on one host until it finishes.

Stand-in: serial versus forked timing

The model below has 12 hosts and four tasks, with slightly different speeds per host. It compares one fork, five forks, twelve forks, the free approximation and skipping fact gathering. The numbers are invented task durations, not measurements of Ansible, but the shape of the result is the point.

JS How forks and strategies change total time

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice three things. Going from one fork to five is a huge win, and five to twelve still helps. With only five forks, the worker pool is the bottleneck, so free gains nothing; with enough forks it finishes when the slowest host finishes. And skipping fact gathering saves a fixed slice of every run.

Long-running tasks: async and poll

By default a task holds its worker and its SSH connection until the module finishes. For something slow, such as a large package upgrade, you can start it in the background and check on it later.

YAML
- name: Start a long upgrade without waiting
  ansible.builtin.apt:
    upgrade: dist
  async: 3600        # allow up to an hour
  poll: 0            # do not wait; return a job id immediately
  register: upgrade_job

- name: Do other work here ...
  ansible.builtin.debug:
    msg: "Upgrade is running in the background"

- name: Wait for the upgrade to finish
  ansible.builtin.async_status:
    jid: "{{ upgrade_job.ansible_job_id }}"
  register: job_result
  until: job_result.finished
  retries: 120
  delay: 30

With poll: 0 the task "fires and forgets", and async_status is how you collect the result. With a positive poll Ansible checks every that many seconds but still waits, which is mainly useful to avoid SSH timeouts on slow tasks. Check mode does not run async tasks.

A word on Mitogen

Mitogen for Ansible is a third-party strategy plugin that replaces the module-copy mechanism with a persistent remote helper process, and in the past it gave large speed-ups. The catch is compatibility: it supports only certain Ansible versions, it is not part of Ansible, and a new Ansible release can break it. Treat it as an optional experiment, check the project's supported versions before adopting it, and measure first. Forks, pipelining and fewer facts usually give most of the benefit with nothing extra to maintain.

Dynamic inventory

In the cloud, servers appear and disappear, so a hand-edited hosts.ini goes stale. A dynamic inventory asks an API at run time. Modern Ansible does this with inventory plugins that you configure with a small YAML file. For Amazon EC2 the plugin is amazon.aws.aws_ec2, and the file name must end in aws_ec2.yml:

YAML
# inventory/aws_ec2.yml
plugin: amazon.aws.aws_ec2
regions:
  - eu-central-1
filters:
  tag:Environment: production
keyed_groups:
  - key: tags.Role
    prefix: role
hostnames:
  - private-ip-address

Here keyed_groups builds groups such as role_web from each instance's Role tag, so your playbooks can say hosts: role_web and never mention an address. Similar plugins exist for Azure, Google Cloud, VMware, Docker and others. Check what the inventory produces before using it:

Terminal
ansible-inventory -i inventory/aws_ec2.yml --graph
ansible-playbook -i inventory/aws_ec2.yml site.yml

Calling a cloud API on every run costs time, so inventory plugins can cache their result (cache: true with a timeout). Keep credentials out of the file; use the provider's normal environment variables or profiles, or Ansible Vault.

Stand-in: an inventory plugin in miniature

An inventory plugin just turns "some records from somewhere" into groups. This imitation takes made-up instance records (what a cloud API would return) and builds groups from tags.

JS Keyed groups from tagged instances

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Add a fifth production instance with Role: "cache" and run again: a new group appears with no other change, which is exactly why dynamic inventory suits changing fleets.

ansible-pull: reverse the direction

Normally the control node pushes to many hosts. With ansible-pull, each host pulls: it fetches a playbook repository from Git and runs it against itself, usually from cron or a systemd timer.

Terminal
ansible-pull -U https://git.example.com/ops/config.git -i localhost, site.yml

This scales without any central machine holding SSH access to everything, and it suits machines that start up on their own (autoscaled instances, laptops, edge devices). The trade-offs: you lose the central view of what ran where, you must protect the repository (whoever can push to it controls every host), and every node needs Ansible installed.

AWX and Ansible Automation Platform

Running ansible-playbook from laptops does not suit a team: who ran what, with which credentials and which version of the playbook? AWX is the open-source web application that answers this, and Red Hat Ansible Automation Platform (AAP) is the supported product built from the same ideas. The main concepts are the same in both:

  • Projects point at a Git repository of playbooks.
  • Inventories can be static or sync from dynamic sources.
  • Credentials hold SSH keys and cloud tokens, stored encrypted and never shown to the person who launches the job.
  • Job templates combine a playbook, inventory and credentials into a button, a schedule, an API call or a webhook target.
  • RBAC, logs and workflows control who may run what, keep an audit trail of every job, and chain templates together.

AWX runs jobs inside execution environments, which are container images holding a specific Ansible version and collections, so "works on my machine" differences disappear. It needs a Kubernetes-style deployment to host, which is real operational work, so many teams start with ansible-playbook in a CI job (see Ansible with Docker, Terraform and CI/CD) and move to AWX or AAP when the audit and access needs appear.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the forks setting control?

    forks is the size of the worker pool, the number of hosts handled at the same time. The default is 5.
  2. 2. Enabling pipelining makes tasks that use become fail on an old server. Which setting is the usual cause?

    Pipelining sends the module through the SSH session to the interpreter, which needs no terminal for sudo. A requiretty rule in sudoers refuses that.
  3. 3. When is strategy: free a bad idea?

    free lets hosts run ahead independently. If step 2 depends on all hosts finishing step 1, such as draining nodes before an upgrade, you need linear (or serial).
  4. 4. What does poll: 0 on an async task do?

    With poll 0 Ansible does not wait. You collect the result later with async_status, using the registered job id.
  5. 5. What is the main idea of ansible-pull?

    ansible-pull reverses the direction: managed hosts fetch the repository and apply it locally, often on a timer.

Summary

  • Total time is roughly hosts times tasks times round trips, divided by how many hosts run at once.
  • Raise forks, turn on pipelining (watch for requiretty), and keep SSH multiplexing.
  • Skip or trim fact gathering (gather_facts: false, gather_subset) or cache facts with gathering = smart.
  • linear waits for every host at each task; free lets hosts run ahead; neither fixes a pool that is too small.
  • async with poll: 0 plus async_status handles very long tasks; Mitogen is a third-party option with version limits.
  • Dynamic inventory plugins build groups from a cloud API; ansible-pull reverses push into pull; AWX and AAP add central control, credentials, scheduling and audit.

Next: Ansible with Docker, Terraform and CI/CD.