Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 36 ยท State and Backends

Refresh and Drift: When the Cloud Changes Behind Pulumi's Back

Understand configuration drift in Pulumi: how pulumi refresh and pulumi up --refresh reconcile state with the cloud, drift detection, ignoreChanges and handling manual edits.

Pulumi believes its state. If you change a security group in the AWS console, Pulumi does not know until it looks. That mismatch is drift: the real infrastructure no longer matches the last recorded state. Drift happens through emergency console fixes, other tools, autoscaling, cloud-side defaults and teammates who "just changed one thing".

This lesson covers how to find drift, how to resolve it in each direction, and when to tell Pulumi to ignore it. The real commands need a cloud account and are read-only here; the stand-in compares three views of a resource so you can practice the decisions.

The three states

Every update involves three versions of the truth:

  1. Program: what your code says should exist (desired).
  2. State: what Pulumi recorded after the last update.
  3. Cloud: what really exists right now.

Normally 1 differs from 2 (you edited code) and 2 equals 3. Drift is when 2 differs from 3.

flowchart LR
  P["Program (desired)"] -- "pulumi up diffs" --> S["State (recorded)"]
  S -- "pulumi refresh reads" --> C["Cloud (actual)"]
  C -- "drift appears here" --> S

up compares program to state; refresh compares state to cloud.

Refreshing

Terminal
pulumi refresh                    # preview changes to state, then apply them
pulumi refresh --preview-only     # show drift without writing state
pulumi refresh --yes              # non-interactive (CI)
pulumi up --refresh               # refresh first, then update in one step
pulumi preview --refresh          # preview that already accounts for drift
pulumi refresh --target urn:...   # only refresh certain resources

refresh calls each provider's read operation for each resource and updates the state to match. It never changes the cloud. It rewrites state, and afterwards the next pulumi up sees the difference between program and the refreshed state.

That has an important consequence. Suppose someone opened port 22 in a security group in the console and your program does not have that rule.

  • After refresh, state contains the extra rule.
  • The next up finds that your program lacks it and removes it from the cloud. This is the desired behavior if the program is the source of truth.
  • If the manual change was right, edit the program to include it before running up.

If you skip the refresh, pulumi up compares program to stale state, sees no difference and leaves the console change in place, silently. That is why many teams run pulumi up --refresh in CI, or schedule refresh --preview-only to report drift.

Refresh also detects deleted resources: a resource removed in the console disappears from state, and the next up recreates it.

Drift detection as a routine

  • CI: run pulumi preview --refresh --expect-no-changes on a schedule. A non-zero exit code means drift or an unapplied change.
  • Pulumi Cloud: Deployments can run scheduled drift detection and optionally remediate automatically, with notifications.
  • Hand work: if a console change was an emergency, record it in code the next business day and run an update so the code is the source of truth again.
Terminal
pulumi preview --refresh --expect-no-changes --diff
echo "exit code: $?"

Stand-in: classify drift

The stand-in takes program, state and cloud snapshots and says what refresh and the following up would do, property by property.

Node.js Stand-in: what refresh and up would do

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the output carefully: the port-22 rule is removed because the program is the source of truth, and the bucket's versioning is switched back on. The queue deleted by hand is simply absent from refreshed state; because it is not in the program in this toy model, nothing is recreated. In a real program the queue would still be declared, so up would create it again.

Choosing how to resolve drift

Situation Action
The manual change was a mistake pulumi up --refresh: Pulumi reverts it
The manual change should stay Edit the program to match, then pulumi up --refresh (no diff left)
A property is managed elsewhere (autoscaler sets desiredCapacity) Add ignoreChanges for it
The resource was deleted by hand and should stay deleted Remove it from the program; refresh already dropped it from state
You cannot tell what changed pulumi refresh --preview-only --diff and read the properties

Some properties are legitimately changed by other systems: a desired count owned by an autoscaler, tags added by a cost tool, a container image deployed by another pipeline. Tell Pulumi to stop fighting:

TypeScript
const asg = new aws.autoscaling.Group("web", {
  minSize: 2,
  maxSize: 10,
  desiredCapacity: 2,
  // ...
}, {
  ignoreChanges: ["desiredCapacity"],
});

ignoreChanges still uses the value on creation but ignores later differences. Use it sparingly and write down why in a comment, because it hides real drift. Related options:

  • protect: true stops accidental deletes.
  • retainOnDelete: true removes the resource from state without deleting it from the cloud.
  • replaceOnChanges: ["prop"] forces a replacement when a property changes.
  • deleteBeforeReplace: true for resources whose names must be unique.

Pitfalls

  • Skipping refresh in CI lets console edits live on invisibly.
  • Refreshing a huge stack is slow and rate-limited: target subsets with --target or split into smaller stacks.
  • Refresh can fail on resources you lack permission to read. Give the CI role read access to everything the stack manages.
  • Some properties are not readable back (write-only passwords). Providers handle this, but you may see spurious diffs; ignoreChanges is the right tool there.
  • Do not refresh with someone else's update running; locking prevents it, so wait.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does pulumi refresh change?

    refresh reads the cloud and rewrites state. It does not modify cloud resources.
  2. 2. Someone added an inbound rule in the console. You run pulumi up --refresh and your program lacks the rule. What happens?

    Refresh records the extra rule, then the update sees the program lacks it and removes it, because the program is the source of truth.
  3. 3. Which option fits a property that an autoscaler legitimately changes after creation?

    ignoreChanges makes Pulumi ignore later differences in the listed properties.
  4. 4. Which command makes a scheduled job fail when the cloud has drifted?

    --expect-no-changes returns a non-zero exit code if the preview shows any difference.

Summary

  • Drift is a mismatch between recorded state and the real cloud; up alone does not see it.
  • pulumi refresh updates state from the cloud; pulumi up --refresh does it and then applies your program.
  • The program is the source of truth: console edits are reverted unless you put them in code.
  • Schedule pulumi preview --refresh --expect-no-changes to detect drift early.
  • Use ignoreChanges for properties owned by another system, and document why.

Next: Import and State Surgery: Adopting Resources and Fixing State