Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท CI/CD and Operations

Safe Operations: Upgrades, Rollbacks and Day-2 Tasks in Pulumi

Run Pulumi safely in production: provider and SDK upgrades, rolling back by redeploying, history, targeted updates, replace and delete options, cancelled updates and retiring stacks.

Writing the first deployment is the easy part. The hard part is the next three years: upgrading providers, undoing a bad change, recovering from a half-finished update, retiring old environments and keeping several people from stepping on each other. This lesson is a field guide to those day-2 operations.

The commands touch real stacks, so they are read-only here. The runnable stand-ins model the version rules and history logic that help you decide.

Rollbacks: Pulumi has no "undo" button

pulumi up applies the program in front of it. There is no built-in rollback, because state records what exists, not a reversible script. To go back:

  1. Revert the code (git revert the commit, or deploy the previous tag).
  2. Run pulumi preview; it should show the reverse of the bad change.
  3. Run pulumi up.

That is why infrastructure code lives in Git and pipelines deploy from tagged commits. Know the limits: a deleted database does not come back by redeploying the old code (it comes back empty), and some changes (for example enabling a feature that cannot be disabled) are irreversible. Use protect, backups and previews to avoid needing a rollback.

pulumi stack history lists past updates (who, when, what changed, which version); Pulumi Cloud adds a UI with diffs. Restoring an old state file with stack import is a last resort for corrupted state, not a deployment rollback, because it makes state disagree with the cloud.

Upgrading providers and SDKs

Provider versions change resource behavior. Upgrade deliberately:

  • Pin versions (lock files: package-lock.json, requirements.txt or go.sum). Upgrades are then a reviewed PR, not a surprise.
  • Read the changelog of the provider for breaking changes, especially major versions (renamed resources, split resources, removed properties).
  • Upgrade one thing at a time: Pulumi CLI, language SDK, provider.
  • Preview in a lower environment first, then prod. A provider upgrade should usually produce no changes. If it shows replacements, stop and investigate.
  • Use pulumi about to print versions of the CLI, language runtime and plugins, and pulumi plugin ls to list installed plugins.
Terminal
npm install @pulumi/aws@latest      # or pin an exact version
pulumi preview --diff               # expect: no changes, or understood changes
pulumi refresh --preview-only       # separate drift from upgrade effects

Major provider upgrades sometimes need a migration step: new resource types replace old ones (for example a bucket configuration split into several resources). The usual technique is to add the new resource with import, remove the old argument and keep the preview empty.

Node.js Stand-in: is this upgrade safe by semantic version?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Targeted operations

Sometimes you need to act on part of a stack.

Terminal
pulumi up --target urn:pulumi:prod::shop::aws:ec2/instance:Instance::web
pulumi up --target-dependents ...        # also dependents of the target
pulumi up --replace urn:...              # force replacement of a specific resource
pulumi destroy --target urn:...          # delete only a part
pulumi up --exclude urn:...              # skip a resource this time

--target is a power tool: it skips the rest of the program, so state and code can diverge, and Pulumi warns about it. Prefer fixing the root problem, and run a full pulumi up soon afterwards. Typical good uses: unsticking a failed resource, forcing replacement of a corrupted instance (--replace) and recovering from an outage while you work on a proper fix.

Useful resource options for operations:

Option Effect
protect Blocks delete and replace
retainOnDelete On delete, forget in state but keep the real resource
deleteBeforeReplace Delete the old resource first (for unique-name resources); causes downtime
replaceOnChanges Replace when listed properties change
ignoreChanges Ignore listed properties when diffing
dependsOn Add explicit ordering
customTimeouts Raise create, update or delete timeouts for slow resources

When an update is interrupted

If a CI job is killed or the laptop dies, the update may be half done and the stack stays locked. State still reflects what actually completed, because it is written after each resource operation (it is a log of progress). To recover:

  1. Make sure nobody is actually running an update.
  2. pulumi cancel (releases the lock; the interrupted update is marked as cancelled).
  3. pulumi refresh to sync state with the cloud, which is especially important if a resource was created but its result never reached state (this can leave "pending operations").
  4. If a pending operation is reported, the CLI offers to clear it after refresh; choose whether the resource was created, then re-run pulumi up.

A model of this is below: state is saved after every step, so a crash leaves a consistent prefix and re-running up continues.

Node.js Stand-in: crash in the middle, then resume

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The second run skips what exists and creates only instance and dns. The dangerous case in real life is a crash between the cloud creating a resource and Pulumi recording it: that is what the pending-operations check and refresh are for.

Retiring things

  • Destroy a stack with pulumi destroy, then remove its record with pulumi stack rm. stack rm on a stack that still has resources refuses unless --force, and --force leaves those resources orphaned in the cloud.
  • Keep production safe: protect the crown jewels, restrict who can run destroy (policies, role permissions) and require approvals in CI.
  • Retain data: use retainOnDelete for resources you want to survive the stack's death, and take a snapshot before destroy.
  • Rename or move a project: use pulumi stack rename and update the project name carefully, or move resources with pulumi state move (see Import and State Surgery).

A day-2 checklist

  • Provider and SDK versions pinned; upgrade PRs preview-clean.
  • Scheduled pulumi preview --refresh --expect-no-changes for drift.
  • protect on stateful resources; backups or versioning on state.
  • Documented runbook for locked stacks and pending operations.
  • Tagged releases, so a rollback is "deploy the previous tag".
  • Everyone knows who may use --target, --replace and destroy in prod.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you roll back a bad infrastructure change in Pulumi?

    Pulumi has no rollback command; you redeploy the previous program and verify with a preview.
  2. 2. A provider minor upgrade shows replacements for several resources in preview. What should you do?

    Unexpected replacements mean behavior changed. Read the changelog and fix or pin before applying.
  3. 3. What is a risk of using --target regularly?

    Targeted updates ignore everything else, so run a full update soon after to restore consistency.
  4. 4. After a crashed CI job leaves a stack locked, what is the correct order?

    Cancel releases the lock; refresh reconciles state with what the cloud actually has before the next update.

Summary

  • There is no rollback command: revert code, preview and apply; know that deleted data does not return.
  • Pin versions, upgrade one component at a time and expect a clean preview from provider upgrades.
  • --target, --replace, --exclude and resource options like protect and retainOnDelete are precise tools; use them deliberately.
  • After an interrupted update: confirm nothing runs, cancel, refresh, then re-run up.
  • Retire stacks with destroy then stack rm, and keep a day-2 checklist and runbook.

Next: Project: A Static Website on Object Storage and a CDN