Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท GitOps Principles

The Reconciliation Loop: Drift, Refresh and Self-Healing

Learn how a control loop keeps a cluster matching Git: observe, diff, act. See how Argo CD detects drift, how refresh and hard refresh differ, and how to ignore fields owned by other controllers.

Kubernetes itself is built from control loops. A Deployment controller watches Deployments and ReplicaSets and creates or deletes pods until the numbers match. Argo CD adds one more loop on top: it watches Git and the cluster and creates or updates objects until they match. This lesson looks at that loop closely, because every behavior you will tune later (sync policy, self-heal, ignore rules, refresh intervals) is a setting of this one idea.

Argo CD cannot run on this site. The runnable code is a stand-in that models the loop in JavaScript so you can see the stages and the effect of different timings.

Observe, compare, act

flowchart LR
  Observe["Observe: read desired from Git, read live from the cluster"] --> Compare["Compare: compute the diff"]
  Compare -->|"no difference"| Idle["Report Synced, wait"]
  Compare -->|"difference"| Policy{"Automated sync?"}
  Policy -->|"yes"| Act["Act: apply changes"]
  Policy -->|"no"| Report["Report OutOfSync, wait for a human"]
  Act --> Observe
  Idle --> Observe
  Report --> Observe

The loop never "finishes". It keeps observing.

Argo CD's observation has two halves with different triggers:

Side How it is observed Latency
Desired (Git) Polling every ~3 minutes, or a webhook, or manual refresh Seconds to minutes
Live (cluster) Kubernetes watches maintained by the application controller Almost immediate

So a drive-by kubectl edit is noticed within moments, but a new commit may wait up to three minutes unless you configure a webhook.

Refresh vs hard refresh vs sync

These three are often confused:

  • Refresh re-reads the Git source and re-renders manifests (using the cache when the commit is unchanged), then recomputes the diff. It changes nothing in the cluster.
  • Hard refresh (argocd app get <app> --hard-refresh) first discards the cached rendered manifests, useful when a Helm dependency or a remote Kustomize base changed without a new commit in your repo.
  • Sync applies the desired state to the cluster. It is the only one that writes.

Self-healing and pruning (preview)

Two policies decide what the act step does:

  • Self-heal re-applies desired state when someone changes live resources by hand.
  • Prune deletes live resources that were removed from Git.

Both are off by default. Sync Policies covers them in depth.

Fields other controllers own

Not every difference is a mistake. A HorizontalPodAutoscaler changes spec.replicas; a service mesh injects sidecars; a mutating webhook adds annotations; status fields are always different. If Git says replicas: 2 and the autoscaler moved it to 7, the app would be OutOfSync forever, and self-heal would fight the autoscaler. The remedy is ignoreDifferences in the Application:

YAML
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

Or, better when the field has a managed owner, omit replicas from the Git manifest altogether. Health Checks and Diffing goes into more patterns such as managedFieldsManagers.

Stand-in: a control loop

This simulation runs ten "ticks". Git changes at tick 2, someone edits the cluster at tick 5, and the autoscaler touches replicas at tick 7. Toggle autoSync, selfHeal and ignoreReplicas to see what each policy changes.

JS Stand-in: reconcile ticks

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In the first run, automated sync applies the commit at tick 2, but the manual edit at tick 5 is only reported, because self-heal is off. In the second run self-heal fixes manual edits, while the autoscaler's replica changes are ignored, so the two controllers do not fight.

Stand-in: how often should it poll?

More frequent polling finds changes faster but costs Git requests. This second stand-in computes the request load for a fleet of Applications that share a few repositories (the repo server caches per repository, so the real load depends on repos, not apps; this model shows the worst case when each app is refreshed separately).

JS Stand-in: polling load

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Troubleshooting the loop

  • App stays OutOfSync after sync: a controller or webhook rewrites a field. Look at the diff, then ignore that field or manage it in Git.
  • App never notices a push: webhooks are not configured and three minutes have not passed; refresh manually.
  • Sync flaps repeatedly: two controllers disagree; self-heal and an autoscaler on replicas is the classic case.
  • Old Helm output after changing a dependency: use a hard refresh.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which side of the comparison is observed almost instantly by Argo CD?

    Live resources are watched continuously; Git changes depend on polling or webhooks.
  2. 2. What does a hard refresh do that a normal refresh does not?

    A hard refresh bypasses the manifest cache; neither refresh type writes to the cluster.
  3. 3. An HPA changes replicas and self-heal keeps resetting it. What is a sensible fix?

    Make the autoscaler the owner of that field, either via ignoreDifferences or by not declaring it in Git.
  4. 4. Is the reconciliation loop finished once the app is Synced?

    A control loop runs continuously so that future drift is noticed too.

Summary

  • Argo CD's loop is observe, compare, act; it never stops.
  • Live state is watched continuously; Git is polled (about three minutes) or pushed via webhook.
  • Refresh recomputes the diff, hard refresh drops caches, sync writes to the cluster.
  • Fields owned by other controllers need ignoreDifferences to avoid fights.
  • Self-heal and prune decide how aggressively the loop acts.

Next: Environments and Promotion