Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Operations and Patterns

Upgrades, Backup and Disaster Recovery

Upgrade Argo CD safely, back up with argocd admin export and import, and rebuild from Git after a disaster, including what Git does not contain.

Argo CD is the tool that deploys your platform, so you must be able to upgrade it without drama and rebuild it quickly if it disappears. The good news: because your desired state is in Git, recovery is mostly "install Argo CD again and point it at Git". The details are what separates a five-minute recovery from a lost afternoon.

What is stored where

Data Lives in In Git?
Application, ApplicationSet, AppProject definitions Kubernetes (CRDs) Should be, via GitOps
Settings: argocd-cm, argocd-rbac-cm, argocd-cmd-params-cm ConfigMaps Should be
Repository credentials, cluster credentials, SSO secrets, argocd-secret Secrets No, never plaintext
Rendered manifest cache, app state cache Redis No (rebuilt on demand)
Application sync history status of Applications No (lost with the object)
The actual workloads Target clusters Not affected by Argo CD going away

Importantly, if Argo CD is deleted, your workloads keep running. Only management stops. The exception is deleting Applications that carry the cascading finalizer: that deletes their resources.

Upgrading

Argo CD publishes minor releases roughly every few months and supports the three most recent minor versions with fixes. Follow this routine:

  1. Read the upgrade notes for every version you skip. Argo CD documents breaking changes per minor version (removed fields, changed defaults, behavior changes of diffing, RBAC, or health checks). Upgrade one minor version at a time.
  2. Take a backup (below).
  3. Try the new version in a non-production instance first.
  4. Apply the new manifests or chart. Large CRDs need server-side apply: kubectl apply --server-side --force-conflicts -f install.yaml, because the CRD annotation can exceed the client-side apply size limit.
  5. Watch the rollout of components, the Applications' sync and health summary, and the controller logs.
  6. Keep the previous manifests so you can roll back. CRD schema changes are generally additive, but check the notes before downgrading.

Pin image versions; never use latest. If you install Argo CD from a Helm chart managed by Argo CD itself (self-management), pin the chart version and upgrade by pull request. Disable prune for the Argo CD application or protect its CRDs, so a mistaken commit cannot remove the CRDs and with them every Application.

Backup with the CLI

Terminal
argocd admin export -n argocd > argocd-backup.yaml

The export contains Applications, AppProjects, ApplicationSets, settings ConfigMaps and the Secrets Argo CD owns (repository and cluster credentials, account tokens). That makes the file sensitive: encrypt it, keep it off shared drives and never commit it.

Restore on a fresh installation:

Terminal
argocd admin import -n argocd - < argocd-backup.yaml

Schedule the export (for example a CronJob with a narrowly scoped ServiceAccount that writes to encrypted object storage). Test the restore regularly; an untested backup is a hope.

Disaster recovery: rebuild from Git

Suppose the management cluster is lost. A sound runbook:

  1. Create a new cluster.
  2. Install Argo CD (same version as before, from the manifests in Git or your bootstrap repo).
  3. Restore secrets: repository credentials, cluster credentials (or re-run argocd cluster add), SSO client secrets, the Sealed Secrets private key if you use it. These are the things Git cannot hold in plaintext. Keep them in a secret manager, recoverable independently of the cluster.
  4. Apply the root Application from the app of apps pattern (or the ApplicationSets).
  5. Watch Applications sync. If workload clusters are untouched, Applications show Synced quickly because their live state already matches Git.
flowchart LR
  A["Lost cluster"] --> B["New cluster"] --> C["Install Argo CD"] --> D["Restore secrets"] --> E["Apply root app"] --> F["Applications reconcile"]

Recovery time is dominated by step 3. Rehearse it.

Also plan for loss of the workload cluster: with Git as truth, you create a new cluster, register it with the same name, and the Applications (or an ApplicationSet cluster generator) deploy to it. Stateful data is a separate matter: back up volumes and databases with your storage tooling.

Rollbacks and history

  • argocd app history shop-web lists deployed revisions; argocd app rollback shop-web <id> redeploys one, but it disables auto-sync first because auto-sync would immediately undo it. The GitOps way is to git revert and let sync do the work.
  • History is capped by revisionHistoryLimit (default 10); it is not a backup.

A simple readiness checklist

  • Argo CD itself is defined in Git and reproducible.
  • Credentials are in a secret manager.
  • A tested backup exists, encrypted, with a retention policy.
  • The runbook lists who can run it, and how long it takes.
  • You know the version you must reinstall (pinned in Git).

Stand-in: backup restore checklist model

The stand-in compares an inventory of what a restore needs with what a backup holds, so you can see gaps before a disaster, in the same spirit as a runbook audit.

Python Stand-in: recovery gap check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens to running workloads when Argo CD is deleted (no cascading finalizers)?

    Argo CD manages resources; it does not host them.
  2. 2. Why does the output of argocd admin export need careful handling?

    Repository and cluster credentials are included, so treat the file as a secret.
  3. 3. How should you upgrade across several minor versions?

    Breaking changes are documented per minor version, so go step by step.
  4. 4. What does argocd app rollback do to auto-sync?

    Rollback deploys an older revision, which auto-sync would immediately replace. The GitOps way is to revert in Git.

Summary

  • Git holds desired state; Secrets, credentials and sync history need separate protection.
  • Upgrade one minor version at a time, back up first, use server-side apply for CRDs, and pin versions.
  • argocd admin export and import back up and restore Argo CD's objects; protect the file.
  • Disaster recovery is install, restore secrets, apply the root app; rehearse it.

Next: Progressive Delivery with Argo Rollouts