Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 38 ยท DevSecOps

Secrets Scanning and Management in Pipelines

Stop credential leaks: how secret scanners find keys with patterns and entropy, pre-commit and push protection, rotating leaked keys, OIDC instead of static keys and secret managers.

A secret is any value that proves identity or grants access: passwords, API keys, cloud access keys, database connection strings, private keys, tokens and certificates. Secrets leak in ordinary ways: a developer commits a .env file, pastes a token into a script, prints credentials in a build log or ships them in a container image. Automated bots scan public repositories continuously, and a leaked cloud key can be abused within minutes.

This lesson covers three jobs: finding secrets before they spread, responding when one leaks, and designing pipelines so there are fewer long-lived secrets to leak in the first place. All keys in the examples are fake. The AWS-style key pair used here is the placeholder pair published in AWS documentation, which does not grant access to anything.

How secrets leak

  • Committed to source control (and left in history after a "fix" commit).
  • Printed in CI logs, error messages or debug output.
  • Baked into container image layers (container image security).
  • Stored in plain environment variables visible to many people or processes.
  • Shared in chat, tickets or wikis.
  • Left in public snapshots, mobile apps and front-end bundles.

Deleting the file in the next commit does not fix it: the old commit remains in the history, in every clone and often in forks and caches. Treat any leaked secret as compromised.

How scanners find secrets

Secret scanners (gitleaks, TruffleHog, GitHub secret scanning, GitGuardian and others) combine three techniques:

  1. Patterns for known formats. Many providers use recognisable prefixes and lengths: AWS access key IDs start with AKIA followed by 16 characters, GitHub tokens start with ghp_, private keys begin with -----BEGIN ... PRIVATE KEY-----.
  2. Entropy analysis. Random-looking strings (high Shannon entropy) near words like secret, token or password are suspicious even without a known format.
  3. Verification. Some tools check whether a candidate is live by calling the provider's API, which separates real leaks from test values. Do this only for credentials your organisation owns.

They also scan history, not just the current files.

Try it: a small secret scanner

This stand-in uses patterns plus an entropy check, and it works on made-up text. It demonstrates why both are needed.

Python Find secrets with patterns and entropy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scanner reports the AWS key ID, the password inside the database URL, the private key header and two high-entropy values next to secret-like names. It ignores the colour code, the "password rules" sentence and the commit hash. Real tools must make the same trade-off between false positives (noise, so developers ignore results) and false negatives (missed leaks), and they use allow-lists for known test values.

Stop leaks early: three layers

flowchart LR
    A["Developer machine<br/>pre-commit hook"] --> B["Git host<br/>push protection"]
    B --> C["CI pipeline<br/>full-history scan"]
    C --> D["Registry and runtime<br/>image and log scans"]

Block at the keyboard, block at the push, scan everything periodically.

A pre-commit hook configuration (read-only example) that runs a scanner on staged changes:

YAML
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.x.x        # pin a reviewed version
    hooks:
      - id: gitleaks

Pre-commit hooks can be skipped, so also enable push protection on the Git host and a CI scan that cannot be bypassed.

When a secret leaks: respond in order

  1. Revoke or rotate first. The secret is compromised the moment it is public. Disable the key, issue a new one and update consumers.
  2. Review usage. Check audit logs for activity with the old key during the exposure window (cloud incident response).
  3. Clean up history only after rotation (tools such as git filter-repo). History rewriting does not undo exposure; it just reduces future copies.
  4. Find the cause. Why was the secret in code? Fix the process, not just the instance.
  5. Add a detection or control so the same class of leak is blocked next time.

Fewer secrets: better designs

The safest secret is one that does not exist. Prefer these approaches:

  • Workload identity and short-lived credentials. Cloud roles attached to compute (instance profiles, managed identities, Kubernetes service accounts mapped to cloud roles) deliver temporary credentials automatically. Nothing is stored in code.
  • OIDC federation for CI/CD. Instead of storing a cloud key in your CI system, the pipeline presents a signed token that identifies the repository and branch, and the cloud exchanges it for temporary credentials. You can restrict a role so that only a given repository and branch can assume it. A read-only sketch of a trust condition:
JSON
{
  "Effect": "Allow",
  "Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/ci.example.com" },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "ci.example.com:sub": "repo:acme/shop:ref:refs/heads/main"
    }
  }
}
  • Secret managers (AWS Secrets Manager, Azure Key Vault, Google Secret Manager, HashiCorp Vault) store, encrypt, audit and rotate the secrets you cannot avoid. Applications fetch them at run time with their own identity.
  • Scoped, separate secrets per environment and per service, so one leak does not open everything.
  • Rotation on a schedule and after staff changes, preferably automated.

Try it: model a pipeline's access decisions

This stand-in evaluates whether a CI job may assume a deployment role, based on trust conditions like those above.

Python Evaluate OIDC trust conditions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only the main branch of the real repository may deploy to production, and a feature branch can reach only the test role. No long-lived key exists anywhere to steal.

Protect secrets inside the pipeline

  • Mask secrets in logs and never echo them; disable shell tracing (set -x) around steps that use them.
  • Limit scope: make secrets available only to the jobs, branches and environments that need them, and require approval for production environments.
  • Treat pull requests from forks as untrusted: do not expose secrets to their workflows.
  • Pin and review third-party actions and plugins, which run with your secrets (supply chain and SBOM).
  • Use ephemeral runners that are destroyed after each job, so nothing persists.
  • Audit access to the secret manager and alert on unusual reads.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A developer commits an API key, then deletes it in the next commit. What should happen?

    Git keeps history, clones and caches. Treat the key as compromised: revoke and replace it, then review usage.
  2. 2. Why combine pattern matching with entropy checks?

    Known formats are matched precisely, while entropy catches custom or unknown token types, at the cost of more false positives.
  3. 3. What is the main benefit of OIDC federation for CI/CD?

    The pipeline gets short-lived credentials by proving its identity, and the role can be limited to one repository and branch.
  4. 4. Which should be done first when a cloud key is found in a public repository?

    Bots scrape public repositories within minutes. Revoking the key is urgent; history cleanup comes afterwards.

Summary

  • Secrets leak through commits, logs, images, environment variables and chat; deleting a file does not undo a leak.
  • Scanners use known patterns, entropy and sometimes verification, on both files and history.
  • Defend in layers: pre-commit hooks, push protection, CI scans and image/log scans.
  • Respond by revoking first, reviewing usage, cleaning history, fixing the cause and adding detection.
  • Prefer workload identity, OIDC federation and secret managers so that fewer long-lived secrets exist, and protect those that remain inside the pipeline.

Next: Supply Chain Security and SBOM