Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 78 ยท Packaging and Delivery

GitOps with Argo CD and Flux

Learn GitOps for Kubernetes: Git as the source of truth, pull-based sync, drift detection, self-heal, app-of-apps and secrets, with a runnable sync loop.

How does a change get from a developer's laptop into a cluster? The oldest answer is that someone runs kubectl apply or helm upgrade by hand. The next answer is a CI pipeline that holds cluster credentials and runs the same commands after every merge. Both work, and both share a weakness: the cluster's real state is whatever the last person or job happened to push, and nobody can say with certainty what is running or why.

GitOps turns this around. You keep the desired state of the cluster in a Git repository, and a program running inside the cluster continuously compares the repository with reality and makes reality match. Git becomes the single source of truth and the audit log: every change is a commit, every rollback is a revert. It is the reconcile loop you already know, with Git as the place where "desired" lives.

Argo CD, Flux and a Git repository cannot run inside this page. Their manifests are read-only, and the Python program is a stand-in that models the core idea (diff, sync, prune, self-heal). The real tools add authentication, caching, health checks, hooks, webhooks and much more.

Four GitOps principles

Declarative The whole system is described as data.
Versioned Git holds history, reviews and rollbacks.
Pulled An agent in the cluster fetches changes.
Reconciled Drift is detected and corrected continuously.
1 / 2

The flow

A GitOps delivery has two repositories in a typical setup. The application repository holds source code, and CI builds and pushes an image. The config repository holds the manifests (plain YAML, Kustomize overlays or Helm values). CI's last step is not "deploy" but "commit the new image tag to the config repository". The agent in the cluster notices the commit and applies it.

flowchart LR
  Dev["Developer"] -->|"merge"| App["App repo"]
  App -->|"CI builds image"| Reg["Registry"]
  App -->|"CI commits new tag"| Cfg["Config repo"]
  Agent["Agent in cluster<br/>(Argo CD or Flux)"] -->|"pulls and compares"| Cfg
  Agent -->|"applies changes"| K8s["Cluster"]
  K8s -->|"live state"| Agent

The agent runs inside the cluster, pulls desired state from Git and compares it with the live state.

Because the agent pulls, the cluster needs no open inbound port for deployments, and your CI system never holds a cluster admin credential.

Argo CD and Flux

Two projects dominate, and both are graduated CNCF projects. They solve the same problem with different shapes.

Argo CD is centered on an Application custom resource and has a rich web UI that shows a live graph of every object and its health. An application points at a Git path and a destination:

YAML
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-web
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/shop-config.git
    targetRevision: main
    path: apps/shop-web/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: shop
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Flux is a set of small controllers (the GitOps Toolkit) that you compose. A GitRepository fetches the source, and a Kustomization or HelmRelease applies it:

YAML
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: shop-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/example/shop-config.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: shop-web
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/shop-web/overlays/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: shop-config
  targetNamespace: shop

Argo CD feels like an application with a dashboard; Flux feels like a set of Kubernetes-native building blocks driven mostly by YAML and the CLI. Pick either; the concepts below apply to both.

Drift, sync, prune and self-heal

Drift is any difference between Git and the cluster. Someone ran kubectl edit during an incident, a controller changed a field, or a commit has not been applied yet. The agent detects drift by comparing the desired manifests with the live objects and reports the application as Synced or OutOfSync.

What happens next depends on policy:

  • Manual sync: the agent shows the diff and waits for a human to press "Sync".
  • Automated sync: new commits are applied without a human.
  • Prune: objects that disappeared from Git are deleted from the cluster. Without prune, deleting a file in Git leaves the object running forever.
  • Self-heal: manual changes to the cluster are reverted. Without self-heal, an automated policy reacts to new commits but leaves hand-made changes alone until the next commit.

Not every difference is a mistake. An autoscaler legitimately rewrites replicas, and admission webhooks add defaults. You tell the agent to ignore such fields (ignoreDifferences in Argo CD, ignore rules in Flux), or the two controllers will fight forever, the HorizontalPodAutoscaler pushing one way and the sync loop pushing back.

The stand-in below plays out a story. Watch the status line and what each policy does or does not do.

Python A GitOps sync loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In step 2 the application is OutOfSync but nothing is reverted, because self-heal is off. In step 3 the replica count differs but is not reported, because it is ignored. Only in step 4 does the loop restore the image. Step 5 shows why people forget prune: the ConfigMap lingers although it is gone from Git. (Real tools only prune objects they created and track by label or annotation, so they never delete things they do not own.)

Structuring the repository

Two patterns are common. A folder per environment (overlays/staging, overlays/prod) lets you promote a change by editing the production folder in a pull request. A branch per environment works too but makes diffs between environments harder to read; most teams prefer folders.

Where a lot of applications exist, you do not want to create each Argo CD Application by hand. The app-of-apps pattern uses one root application whose path contains other Application manifests. Adding a new app means committing one more file; the root picks it up. Argo CD's ApplicationSet generates applications from a template, a list of clusters or a directory listing. Flux achieves the same with a root Kustomization that points to a folder of further Kustomization objects. Large installs also need an order: CRDs and namespaces before the things that use them. Argo CD handles this with sync waves (the annotation argocd.argoproj.io/sync-wave, lowest number first), and Flux with dependsOn.

Python App-of-apps, sync waves and promotion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Secrets in GitOps

If everything lives in Git, where do passwords go? Not in plain text: a base64-encoded Secret is readable by anyone who can read the repository, and Git history keeps it forever. There are three families of solutions, covered in External Secrets, Sealed Secrets and Config Patterns:

  • Encrypt in Git. Sealed Secrets (a controller decrypts a SealedSecret that only your cluster can read) or SOPS (encrypts values in the file; Flux has built-in SOPS decryption, and Argo CD uses a plugin).
  • Reference, do not store. The External Secrets Operator or the Secrets Store CSI driver pulls values from a vault or cloud secret manager at run time. Git holds only a pointer.
  • Keep secrets out of the loop and create them by another process. This is the weakest option because it breaks the "Git describes everything" promise.

Trade-offs and habits

GitOps is not free. Changes are no longer instant (the agent polls every minute or so unless a webhook nudges it), emergency fixes need a commit, and the repository's permissions become your production permissions, so protect the main branch and require reviews. In return you get a reviewable history, easy rollback, a cluster you can rebuild from Git after a disaster, and a stop to "who changed this?" mysteries.

Good habits: let CI commit image tags (never latest), keep one place for each setting, ignore only fields owned by another controller, and make rollbacks boring by using git revert.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What makes a GitOps delivery pull-based?

    In GitOps the agent runs in the cluster and pulls from the repository, so CI needs no cluster credentials and no inbound access is required.
  2. 2. An automated sync policy is on, but self-heal is off. Someone edits a Deployment by hand. What happens?

    Automated sync reacts to new commits. Reverting manual cluster changes is the job of self-heal.
  3. 3. You delete a manifest from Git but its object keeps running in the cluster. What is the most likely cause?

    Without prune the agent never deletes objects that are no longer in Git, so they stay behind.
  4. 4. Why do GitOps tools offer ignore rules for fields such as replicas?

    If two controllers keep overwriting the same field, they flip it back and forth. Ignoring the field in the diff gives ownership to the autoscaler.

Summary

  • GitOps keeps desired state in Git and uses an in-cluster agent (Argo CD or Flux) to pull it and reconcile the cluster continuously.
  • Pull-based delivery keeps cluster credentials inside the cluster; CI only commits a new image tag.
  • Drift is detected by diffing Git with live state; automated sync handles new commits, self-heal reverts manual changes, prune removes deleted objects.
  • Ignore fields owned by other controllers, such as autoscaled replica counts.
  • App-of-apps (or ApplicationSets) scale the pattern to many applications; sync waves or dependsOn order them.
  • Secrets must be encrypted in Git or referenced from an external store, never stored as plain base64.

Next, learn how the loops behind all of this work: Controllers and the Reconcile Loop.