Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Sync and Health

Argo CD Health Checks and Diffing: Healthy, Degraded and Ignored Fields

Learn how Argo CD computes resource and application health, what Healthy, Progressing, Degraded, Suspended and Missing mean, how to write custom Lua health checks and how to tame noisy diffs.

An Application has two independent verdicts. Sync status asks "does the cluster match Git?". Health status asks "does what is running actually work?". This lesson explains both verdicts, how health is computed for each kind of resource and what to do when either one lies to you, whether because a custom resource has no health logic or because another controller keeps changing fields.

The health rules and YAML are real, but cannot run on this site. The runnable blocks are stand-ins: a health aggregator and a diff normalizer. The Lua health script shown is read-only because Argo CD evaluates it itself.

Health statuses

Status Meaning
Healthy The resource is fully operational.
Progressing Not healthy yet, but expected to become so (a rollout in progress, a pod starting).
Degraded The resource failed or timed out (crash loop, failed Job, exceeded progress deadline).
Suspended Paused on purpose (a CronJob or a paused Rollout).
Missing The resource is in Git but does not exist in the cluster.
Unknown Health could not be determined.

The Application's health is the worst of its resources' health values (with some nuance for ignored kinds), so one crash-looping pod makes the whole app Degraded.

Built-in health logic

Argo CD ships checks for common kinds:

  • Deployment, StatefulSet, DaemonSet, ReplicaSet: compare desired and ready/updated replicas and the rollout conditions.
  • Service: a LoadBalancer is Progressing until it has an ingress address.
  • Ingress: Progressing until a load balancer address is set.
  • PersistentVolumeClaim: Healthy when Bound.
  • Pod: based on phase, container states and restart behavior.
  • Job: Progressing while running, Degraded when failed, Healthy when complete.
  • Application: an app of apps is as healthy as its child Applications.

Kinds with no check (most custom resources) are reported as Healthy as soon as they exist, which can hide problems.

Custom health checks in Lua

For a custom resource, add a Lua script to the argocd-cm ConfigMap under resource.customizations.health.<group>_<kind>:

YAML
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.cert-manager.io_Certificate: |
    local hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for _, c in ipairs(obj.status.conditions) do
        if c.type == "Ready" and c.status == "False" then
          hs.status = "Degraded"
          hs.message = c.message
          return hs
        end
        if c.type == "Ready" and c.status == "True" then
          hs.status = "Healthy"
          hs.message = c.message
          return hs
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

obj is the live resource; the script returns a table with status and message. Many popular CRDs (cert-manager, Argo Rollouts, Crossplane and others) already ship with checks in Argo CD's built-in set.

Stand-in: aggregate health

This model combines resource health values into an application verdict and lists the culprits.

JS Stand-in: health aggregation

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Diffing: why is it OutOfSync?

Argo CD diffs the rendered manifest against the live object, after normalizing both (it removes status, defaulted fields it knows about and some metadata). Remaining differences come from:

  • mutating admission webhooks (injected sidecars, default annotations),
  • controllers and autoscalers editing spec fields,
  • API defaulting for fields Argo CD does not know,
  • real drift.

ignoreDifferences

On the Application:

YAML
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - '.spec.template.spec.containers[] | select(.name == "istio-proxy")'
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions:
        - '.webhooks[]?.clientConfig.caBundle'
    - kind: Service
      managedFieldsManagers:
        - cloud-controller-manager
  • jsonPointers ignore an exact path.
  • jqPathExpressions select paths with jq syntax, handy for list items.
  • managedFieldsManagers ignore every field last written by the named manager, a neat way to say "the autoscaler owns this".

Add the sync option RespectIgnoreDifferences=true so that a sync does not overwrite those fields anyway. For rules that apply to every app, set resource.customizations.ignoreDifferences.<group_kind> in argocd-cm.

Stand-in: a diff normalizer

This model removes paths from both objects before comparing them, the way ignoreDifferences does. Edit the ignored list to see the effect.

JS Stand-in: ignoreDifferences

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Reading the diff in practice

Use argocd app diff <name> or the Diff tab. If a resource stays OutOfSync right after a sync, the diff almost always names the field a controller keeps rewriting. Decide who should own that field: Git (then stop the other writer) or the other controller (then ignore it).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A Deployment's manifests match Git, but its pods crash-loop. What are the sync and health statuses?

    Sync compares configuration; health reflects behavior. The pods failing makes health Degraded.
  2. 2. How is an application's health derived from its resources?

    One degraded resource makes the Application degraded.
  3. 3. What does managedFieldsManagers in ignoreDifferences do?

    It uses server-side managedFields to ignore changes owned by a particular controller.
  4. 4. Which language are custom health checks written in?

    Custom health checks are Lua scripts in argocd-cm.

Summary

  • Health (does it work?) is separate from sync (does it match Git?).
  • Statuses: Healthy, Progressing, Degraded, Suspended, Missing, Unknown; the app takes the worst.
  • Built-in checks cover core kinds; add Lua checks for custom resources.
  • Noisy diffs come from controllers, webhooks and defaults; use ignoreDifferences with RespectIgnoreDifferences.
  • Decide who owns each field, Git or another controller.

Next: Sync Hooks and Rollbacks