Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Operations and Patterns

Notifications and Git Webhooks

Send Slack, email and webhook alerts on sync and health events with Argo CD Notifications, and use Git webhooks for instant refresh instead of polling.

Two small features make Argo CD feel much more responsive and observable: Git webhooks, which tell it about a new commit immediately, and Notifications, which tell people when something succeeded, failed or became unhealthy.

Polling versus webhooks

By default the application controller refreshes each Application on a timer: timeout.reconciliation in argocd-cm, three minutes by default (with some jitter). A merge can take up to three minutes to show up. Lowering the timer increases load on the repo server and the Git provider.

A webhook removes the wait. The Git provider calls Argo CD on every push; Argo CD finds the Applications that use that repository and refreshes them at once.

sequenceDiagram
  participant D as Developer
  participant G as Git provider
  participant A as Argo CD API server
  participant C as Application controller
  D->>G: git push
  G->>A: POST /api/webhook
  A->>C: Refresh Applications for that repo
  C->>C: Compare and sync

A webhook turns "within three minutes" into "within seconds".

Configure it in your Git provider: URL https://argocd.example.com/api/webhook, content type application/json, event "push" (add pull request events if you use PR generators). Add a shared secret to the provider and to argocd-secret (keys such as webhook.github.secret, webhook.gitlab.secret) so forged calls are ignored. GitHub, GitLab, Bitbucket, Azure DevOps, Gitea and others are supported.

You can also trigger a refresh yourself:

Terminal
argocd app get shop-web --refresh
argocd app get shop-web --hard-refresh   # also drops the manifest cache

Keep polling as a safety net: webhooks can be lost.

What Notifications is

Notifications is a controller (bundled in current installs) that watches Applications and sends messages when a trigger becomes true, formatted by a template, delivered through a service.

Concept Meaning
Service Where to send: Slack, Microsoft Teams, email, a generic webhook, GitHub commit status, Opsgenie, PagerDuty and more
Template The message body, written with Go templates over the Application
Trigger A condition plus the template to send, such as "sync failed"
Subscription Who gets it, set by an annotation on the Application or project

Configuration

Everything lives in argocd-notifications-cm, and credentials in argocd-notifications-secret:

YAML
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  template.app-sync-failed: |
    message: |
      Application {{ .app.metadata.name }} failed to sync.
      {{ .app.status.operationState.message }}
    slack:
      attachments: |
        [{"color": "#E96D76", "title": "{{ .app.metadata.name }}",
          "text": "{{ .app.status.operationState.phase }}"}]
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]
YAML
apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
stringData:
  slack-token: <token-from-secret-manager>

The catalog of ready-made triggers and templates (on-deployed, on-health-degraded, on-sync-failed, on-sync-running, on-sync-status-unknown, on-sync-succeeded) can be installed from the project's catalog manifest, which saves you from writing them by hand.

Subscribing

Subscriptions are annotations: notifications.argoproj.io/subscribe.<trigger>.<service>: <recipient>.

YAML
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-failed.slack: shop-alerts
    notifications.argoproj.io/subscribe.on-health-degraded.slack: shop-alerts

Subscribe at the AppProject level to cover every Application in it, or define subscriptions with defaultTriggers in the ConfigMap to cover the whole installation. In ApplicationSet templates, put the annotations in the template so every generated app is covered.

Notifying Git: commit status

A useful trick: send the sync result back to the commit as a status check, so a pull request or the commit list shows whether the change actually deployed. Use the GitHub or GitLab service with a template that includes the revision ({{ .app.status.operationState.operation.sync.revision }}).

Debugging notifications

Terminal
argocd admin notifications template notify app-sync-failed shop-web
argocd admin notifications trigger run on-sync-failed shop-web

These dry-run a template or trigger against a real Application. Also read the notifications controller logs. A message sent twice usually means two triggers fire; Notifications remembers what it has sent in an annotation on the Application, so it does not repeat for the same state.

Alert on meaning, not on every event. on-sync-failed and on-health-degraded deserve a channel; on-sync-succeeded for every app of a large fleet is noise people will mute.

Stand-in: evaluating triggers

This model checks trigger conditions against application states and renders a message, so you can see why a trigger fires. Real trigger conditions use the Expr language; the stand-in uses Python functions.

Python Stand-in: triggers and templates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the default interval at which Argo CD refreshes Applications from Git?

    timeout.reconciliation defaults to 180 seconds (with jitter).
  2. 2. Which endpoint receives Git provider webhooks?

    Point the provider's webhook at the API server's /api/webhook path.
  3. 3. How does an Application subscribe to a Slack alert on failure?

    Subscriptions are annotations naming the trigger, the service and the recipient.
  4. 4. Where do notification service credentials belong?

    The ConfigMap references secret keys, so tokens stay in a Secret.

Summary

  • Webhooks to /api/webhook give near-instant refresh; keep polling as a fallback.
  • Notifications combine services, templates, triggers and annotation-based subscriptions.
  • Store tokens in argocd-notifications-secret and subscribe at project level for coverage.
  • Alert on failures and degraded health, not on every success.

Next: High Availability and Performance