Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Operations and Patterns

Troubleshooting Sync Problems

Diagnose common Argo CD problems: permanent OutOfSync, comparison errors, stuck operations, finalizers, hooks and ignoreDifferences, with commands to find the cause.

When something goes wrong in Argo CD, the cause is nearly always one of a small number of patterns. This lesson gives you a method first, then the recurring problems and their fixes.

A method

  1. Read the status. argocd app get <app> shows sync status, health, the last operation's message and per-resource results.
  2. Find the layer. Is it rendering (the repo server cannot produce manifests), comparing (live vs desired differ in a way you did not expect) or applying (Kubernetes rejected the change)?
  3. Reproduce locally. Render with the same tool and version (helm template, kustomize build) and compare with argocd app diff <app>.
  4. Check logs of the component for that layer.
flowchart TD
  S["Symptom"] --> Q{"Which layer?"}
  Q -- "ComparisonError, manifest generation" --> R["Repo server logs; render locally"]
  Q -- "OutOfSync that will not clear" --> D["argocd app diff; ignoreDifferences"]
  Q -- "Sync failed, resource error" --> A["Operation message; kubectl describe the resource"]
  Q -- "Operation hangs" --> H["Hooks, finalizers, health of resources"]

Pick the layer, then dig.

Useful commands

Terminal
argocd app get shop-web                    # status, conditions, resources
argocd app diff shop-web                   # desired vs live
argocd app get shop-web --hard-refresh     # drop manifest cache, re-render
argocd app manifests shop-web              # what Argo CD thinks it should apply
argocd app history shop-web                # past syncs
argocd app terminate-op shop-web           # cancel a stuck operation
kubectl logs -n argocd deploy/argocd-repo-server
kubectl logs -n argocd statefulset/argocd-application-controller

Problem: permanently OutOfSync

The app syncs successfully but returns to OutOfSync in seconds. Causes:

  • Controllers mutating resources: an HPA changes replicas, an operator adds defaults, a webhook injects sidecars or annotations, cert-manager rewrites a field. Argo CD sees the difference each time.
  • Defaulted fields: the API server fills fields you did not write. Argo CD normalizes many, but CRDs with defaults can show noise.
  • Non-deterministic rendering: random values or timestamps in charts.

Fix by telling Argo CD to ignore the fields owned by something else:

YAML
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions:
        - '.webhooks[]?.clientConfig.caBundle'
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

Without RespectIgnoreDifferences=true, an ignored field is still overwritten when a sync happens. Add the option so syncs also leave it alone. Server-Side Apply (ServerSideApply=true) often reduces this noise by tracking field ownership.

Problem: ComparisonError

The Application shows a condition like ComparisonError: rpc error ... manifest generation error. The render failed. Typical reasons: a Helm template error, a missing value file, an invalid YAML, an unreachable repository or a missing credential. The message includes the tool's error text; reproduce it locally. If the repository is private, check argocd repo list for a connection status.

Problem: sync fails on specific resources

The operation message names the resource and Kubernetes' reason:

  • invalid: field is immutable: you changed an immutable field (Deployment selector, Job spec, Service clusterIP). Use the Replace=true sync option on that resource, or the sync option Force=true to delete and recreate. Understand the downtime this causes.
  • CustomResourceDefinition ... not found: a custom resource is applied before its CRD exists. Use a sync wave for the CRD, or the SkipDryRunOnMissingResource=true annotation on the custom resource.
  • namespace not found: add CreateNamespace=true to sync options.
  • Forbidden: the Argo CD ServiceAccount (or the project, or RBAC on the target cluster) lacks permission, or the project forbids the kind.
  • Annotation too long: very large resources (CRDs) exceed the client-side apply annotation limit; enable ServerSideApply=true.
  • Webhook timeouts: a validating webhook for a component that is not ready yet. Use sync waves to install that component first.

Problem: the sync hangs

  • Hooks: a PreSync Job that never finishes blocks the whole sync. Inspect the Job and its pods; add activeDeadlineSeconds. Make hooks idempotent and give them a delete policy (argocd.argoproj.io/hook-delete-policy: HookSucceeded).
  • A resource that never becomes healthy: a waiting wave stays blocked on an unhealthy resource. Read the Deployment's events.
  • Stuck operation: argocd app terminate-op <app>, then fix and resync.
  • Retries: with syncPolicy.retry the controller keeps retrying with backoff; look at the "Sync Status" history for the repeating error.

Problem: Application will not delete

An Application with the resources-finalizer.argocd.argoproj.io finalizer waits until its resources are deleted. If a resource's own finalizer is stuck (for example a namespace waiting for a Custom Resource whose controller is gone), deletion hangs. Find what is blocking: kubectl get all -n <ns>, then examine finalizers on the stuck object. Removing finalizers by patching is a last resort and may orphan external resources.

Problem: wrong project, repo or destination

Errors such as application destination is not permitted in project or application repo is not permitted come from the AppProject. Compare the Application's source.repoURL and destination with the project's sourceRepos and destinations (globs allowed).

Problem: shared resource warnings

SharedResourceWarning means the same resource appears in two Applications. Argo CD tracks ownership with a label or annotation; two owners will fight. Give every resource exactly one owner.

Resource tracking

Argo CD marks what it owns with a tracking annotation (argocd.argoproj.io/tracking-id) by default in current versions (older installs used a label app.kubernetes.io/instance). Charts that also set that label cause confusing clashes; keep the annotation method.

Stand-in: a drift and ignore simulator

The stand-in applies "ignore" rules to the difference between desired and live objects, so you can see which fields keep an app OutOfSync.

Python Stand-in: diff with ignoreDifferences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. An HPA changes replicas and the app is OutOfSync again after every sync. What is the usual fix?

    Another controller owns the field, so Argo CD should ignore it.
  2. 2. Which sync option helps with a field is immutable error, at the cost of recreating the resource?

    Replace or Force deletes and recreates the object, which can cause downtime.
  3. 3. A custom resource fails because its CRD does not exist yet. What helps?

    Install the CRD first or skip the dry run for that resource.
  4. 4. Which command cancels a stuck sync operation?

    terminate-op stops the running operation without deleting the Application.

Summary

  • Identify the layer first: rendering, comparing or applying; reproduce locally with app diff and the render tool.
  • Fix endless OutOfSync with ignoreDifferences plus RespectIgnoreDifferences, or server-side apply.
  • Order CRDs, namespaces and webhooks with sync waves and sync options.
  • Unstick operations with terminate-op, and check hooks and finalizers.

Next: Project: Multi-environment Promotion