Skip to content
Tutorials Center
My learning

Topics

Lesson 70 of 78 ยท Projects

Project: CI/CD Pipeline to Kubernetes

Build a pipeline that tests, scans and pushes an image, updates a config repo and lets Argo CD sync and roll back, with runnable promotion and sync models.

Shipping by running kubectl apply from a laptop does not scale. Nobody knows what is really deployed, nobody can review the change, and the person with cluster credentials becomes a bottleneck. In this project you design the flow that most teams settle on: a CI pipeline builds and checks an image, a config repository records which version should run in each environment, and Argo CD makes the cluster match that repository.

Pipelines and clusters cannot run on this site, so the workflow, manifests and commands are read-only and written to be correct, not to be executed here. The runnable parts are Python stand-ins: a pipeline that stops at the first failing gate, a promotion flow with history and rollback, and a sync loop that detects drift. They model the rules, not the real tools. For the image-building half in more detail, read CI Pipelines for Images; for the delivery half, GitOps with Argo CD and Flux is the background.

Two halves of delivery

CI Test, build, scan, push an image.
CD Make the cluster match Git.

CI never needs cluster credentials in this design.

1 / 2
flowchart LR
  A["App repo<br/>push to main"] --> B["CI: test, build, scan"]
  B --> C["Registry<br/>image:sha"]
  B --> D["Config repo<br/>new image tag"]
  D --> E["Argo CD"]
  E --> F["Cluster"]
  F -. "reports health" .-> E

Code flows left to right; Argo CD pulls from Git and pushes into the cluster, so CI never touches the cluster.

Why two repositories?

The app repo holds source code and the Dockerfile. The config repo holds the manifests (or Helm values or Kustomize overlays, see Kustomize) for each environment. Splitting them has real benefits.

  • Every deployment is a Git commit: reviewable, attributable and revertible.
  • A config change (such as more replicas) does not trigger an image build, and a code change does not need to touch manifests by hand.
  • The cluster credentials live with Argo CD inside the cluster. A compromised CI job can push an image, but it cannot run commands against production.

Small teams sometimes keep both in one repo with separate folders. That works, but guard against loops: a commit that changes only the manifest folder must not start another build.

The CI pipeline

The pipeline below uses GitHub Actions syntax, but GitLab CI, Jenkins or any other system can do the same four things. In real use, pin third-party actions to a full commit hash instead of a tag, as supply-chain attacks have targeted mutable tags.

YAML
name: ci
on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write

env:
  IMAGE: ghcr.io/example/shop-web

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run unit tests
        run: make test

      - uses: docker/setup-buildx-action@v3

      - name: Build image (not pushed yet)
        uses: docker/build-push-action@v6
        with:
          context: .
          load: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Scan for vulnerabilities
        uses: aquasecurity/[email protected]
        with:
          image-ref: ${{ env.IMAGE }}:${{ github.sha }}
          severity: CRITICAL,HIGH
          ignore-unfixed: true
          exit-code: "1"

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Push image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}
          cache-from: type=gha

  update-config:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          repository: example/shop-config
          token: ${{ secrets.CONFIG_REPO_TOKEN }}

      - name: Point dev at the new image
        working-directory: envs/dev
        run: |
          kustomize edit set image ${{ env.IMAGE }}=${{ env.IMAGE }}:${{ github.sha }}
          git config user.name "ci-bot"
          git config user.email "[email protected]"
          git commit -am "dev: shop-web ${{ github.sha }}"
          git push

A few decisions in this file are deliberate. The image is tagged with the commit SHA, an immutable identifier, never with latest. A tag that moves makes "what is running?" unanswerable and can make Kubernetes skip an update because the manifest text did not change. The scan runs before the push, so a vulnerable image never reaches the registry. And the update-config job runs only if build succeeded, because of needs.

The first stand-in shows the same gating logic: stages run in order, the first failure stops the run, and only a fully green run touches the config repo.

Python A pipeline that stops at the first failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Promotion between environments

New builds land in dev automatically. Moving the same image to staging and production is a promotion: copy the tag from one environment folder to the next. The key principle is to promote the artifact, never rebuild it. The image tested in staging must be bit-for-bit the one that reaches production.

A common setup promotes to staging automatically after smoke tests pass, while production needs a pull request that a human approves. Because the config repo is just Git, the audit trail comes for free.

Rollback is equally plain: git revert the commit that changed the tag. You will see this in the next stand-in, which keeps a history of config commits.

Python Promotion with history and rollback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The revert restores the whole state from before that commit, so production returns to the earlier tag. Notice that nobody ran anything against the cluster; the history only holds desired state.

Argo CD closes the loop

Argo CD runs inside the cluster, watches a path in the config repo and applies what it finds. One Application object per environment tells it where to look:

YAML
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-web-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/shop-config.git
    targetRevision: main
    path: envs/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: shop
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

automated means that Argo CD syncs whenever Git changes, prune deletes objects that disappeared from Git, and selfHeal reverts manual changes made with kubectl. Typical commands for checking and steering it:

Terminal
argocd app get shop-web-prod
argocd app diff shop-web-prod
argocd app sync shop-web-prod
argocd app history shop-web-prod

A word of caution on rollbacks: argocd app rollback only works while automated sync is off, because otherwise Argo CD would immediately sync back to Git. With GitOps the normal rollback is a Git revert, as in the stand-in above. Secrets should not appear in this repo in plain text; see External Secrets, Sealed Secrets and Config Patterns.

The last stand-in is a sync loop that compares Git with the live cluster, applies the difference and heals drift.

Python Drift detection and self-heal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Hardening the pipeline

Before this goes to production, add the usual safeguards: protect the main branch and require reviews, use short-lived OIDC credentials for the registry instead of long-lived passwords, sign images and verify the signature at admission (Supply Chain and Cluster Hardening), run smoke tests against staging before promotion, and use progressive delivery (Deployment Strategies) so a bad version reaches only a fraction of users. Keep an eye on the whole path with alerts on failed syncs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why tag images with the commit SHA instead of latest?

    An immutable tag makes every deployment traceable and reversible. A moving tag hides which build runs and may not even trigger an update.
  2. 2. What does it mean to promote an image from staging to production?

    Promotion moves the same artifact forward by changing the tag in the production configuration. Rebuilding could produce a different image than the one you tested.
  3. 3. With automated sync enabled in Argo CD, what is the normal way to roll back?

    Git is the source of truth. Reverting the commit changes the desired state and Argo CD syncs to it. The rollback command only works while automated sync is off.
  4. 4. What does selfHeal do?

    selfHeal compares live objects with Git and undoes manual drift, such as a kubectl scale, so the repository stays the single source of truth.

Summary

  • CI builds, tests, scans and pushes an image tagged with the commit SHA, then records the new tag in a config repository.
  • Argo CD runs in the cluster, pulls the config repo and syncs; CI needs no cluster credentials.
  • Promote the same image through environments by changing tags in Git, with approval for production.
  • Roll back by reverting a Git commit; prune and self-heal keep the cluster equal to Git.
  • The Python models show the gating, history and sync logic, while real pipelines and Argo CD run outside this site.

Next, extend Kubernetes itself: Project: A Tiny Operator.