Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Operations and Patterns

High Availability and Performance Tuning

Run Argo CD in HA mode and scale it: component roles, controller sharding, repo server limits, monorepo path annotations, Redis and resource sizing.

A single Argo CD instance can manage hundreds of clusters and thousands of Applications, but only if you understand which component does which work and where the bottlenecks are. This lesson maps the architecture to the knobs you can turn.

The components

flowchart LR
  U["Users, CLI, CI"] --> S["API server"]
  S --> R["Redis"]
  S --> RS["Repo server"]
  C["Application controller"] --> RS
  C --> R
  C --> K["Target clusters"]
  RS --> G["Git and chart repos"]
  AS["ApplicationSet controller"] --> S
  N["Notifications controller"] --> S

What talks to what.

Component Kind Job Scales by
argocd-server Deployment API, UI, auth, webhooks More replicas (stateless)
argocd-repo-server Deployment Clones repos, renders Helm/Kustomize/plugins More replicas, more CPU, parallelism limits
argocd-application-controller StatefulSet Compares and syncs, watches clusters Sharding across replicas
argocd-redis (or Redis HA) Deployment Cache of rendered manifests and app state HA setup
argocd-applicationset-controller Deployment Generates Applications Leader election, one active
argocd-notifications-controller Deployment Sends alerts One active
argocd-dex-server Deployment SSO broker Replicas if used

HA installation

The project ships two manifest sets: the standard install.yaml and an HA variant (ha/install.yaml) that uses Redis HA with Sentinel, multiple replicas of the server and repo server, and anti-affinity rules. HA needs at least three nodes for Redis HA to place its pods apart. Use a Helm chart or Kustomize on top of these and manage the installation itself with GitOps, as described in Projects and Practice.

HA protects against pod and node failures. It does not protect against deleting the namespace or losing the cluster; that is backup and disaster recovery.

Where load comes from

  1. Repo server: rendering is CPU-heavy (Helm, Kustomize) and memory-heavy for big repositories. Each Application refresh may need a render if the cache missed.
  2. Application controller: keeps a cache of live resources for every managed cluster and runs comparisons. Memory grows with the number of resources.
  3. Git provider: many fetches can hit rate limits.
  4. Kubernetes API of managed clusters: watches and list calls.

Application controller sharding

One controller replica handles all clusters by default. With sharding, each replica takes responsibility for a subset of clusters:

YAML
# StatefulSet argocd-application-controller
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: argocd-application-controller
          env:
            - name: ARGOCD_CONTROLLER_REPLICAS
              value: "3"

The replica count and ARGOCD_CONTROLLER_REPLICAS must match. A cluster is assigned to a shard by the sharding algorithm (legacy hash-based, round-robin, or consistent-hashing which limits movement when replicas change). Choose it with controller.sharding.algorithm in argocd-cmd-params-cm. Because sharding is per cluster, one huge cluster cannot be split; give it more memory instead.

Tuning processors

In argocd-cmd-params-cm:

Setting Default What it does
controller.status.processors 20 Parallel app status (comparison) workers
controller.operation.processors 10 Parallel sync operations
controller.repo.server.timeout.seconds 60 Wait for manifest generation
reposerver.parallelism.limit 0 (unlimited) Concurrent manifest generations per repo server
controller.self.heal.timeout.seconds 5 Delay before self-heal re-syncs

A rule of thumb in the docs for sizing: about 20 status processors and 10 operation processors per 1000 Applications. Raise them gradually and watch memory.

Reducing repo server work

  • manifest-generate-paths annotation: in a monorepo every commit touches the repo, so every Application sees "a new revision". The annotation tells Argo CD which paths affect an application, so unrelated commits reuse the cache:
YAML
metadata:
  annotations:
    argocd.argoproj.io/manifest-generate-paths: .;../shared

. is the application's own path; ../shared is a path relative to it. Combined with webhooks, only the Applications that actually changed re-render.

  • Bigger polling interval: with working webhooks, raise timeout.reconciliation to 5 to 10 minutes.
  • Replicas and a shared cache: multiple repo servers share manifests through Redis.
  • Avoid slow renders: huge umbrella charts and Jsonnet loops can take tens of seconds. Set reposerver.default.cache.expiration appropriately and keep render time visible in metrics.
  • Keep repositories lean: large binary history slows every clone, so keep binaries out of the GitOps repository.

Monitoring

Argo CD exposes Prometheus metrics on each component (the controller, server and repo server). Useful ones:

  • argocd_app_info (labels for sync and health status), to alert on Degraded or long OutOfSync.
  • argocd_app_reconcile (histogram) for comparison time.
  • argocd_git_request_total and argocd_git_request_duration_seconds for Git trouble.
  • workqueue_depth of the controller queues; a growing queue means more processors or shards are needed.
  • argocd_cluster_api_resource_objects and cache sizes for memory planning.

Ship them to Prometheus (a ServiceMonitor is provided in the manifests) and build alerts from them, with a Grafana dashboard from the project's examples.

Resource requests and limits

Set requests and limits for every component, and give the controller memory headroom proportional to the number of live resources it caches. An out-of-memory kill of the controller shows up as "resources not updating" and a restarting StatefulSet pod.

Stand-in: sharding and capacity model

This stand-in computes how clusters spread across shards with simple modulo assignment (a model of the idea, not Argo CD's exact hash) and estimates processors for a given Application count.

Python Stand-in: shard assignment and processors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which component is sharded across replicas?

    The controller StatefulSet divides managed clusters among its replicas.
  2. 2. What does the manifest-generate-paths annotation help with?

    In a monorepo it avoids refreshing every Application on every commit.
  3. 3. What must match the StatefulSet replica count when sharding?

    The environment variable tells each replica how many shards exist.
  4. 4. Which component does heavy CPU work rendering Helm and Kustomize?

    The repo server clones repositories and generates manifests.

Summary

  • Know the roles: server (API), repo server (render), controller (compare and sync), Redis (cache).
  • Use the HA manifests, then shard the controller by cluster and scale the repo server.
  • Tune processors and parallelism gradually; use webhooks and manifest-generate-paths to cut needless work.
  • Monitor with Prometheus metrics and set resource limits.

Next: Upgrades, Backup and Disaster Recovery