Skip to content
Tutorials Center
My learning

Topics

Lesson 70 of 111 ยท Security

Managing Secrets

Keep passwords and API keys out of images and logs: why ENV and ARG leak, how BuildKit secret mounts and Compose secrets work, runtime injection and rotation.

A secret is any value that would hurt you if a stranger read it: database passwords, API keys, tokens, private keys, signing keys. Containers make secrets easy to leak, because an image is a stack of shareable layers, docker inspect prints the whole configuration, and environment variables are copied into logs, crash reports and child processes.

This lesson starts with the ways secrets leak, with real output you can reproduce, then shows the safe options: BuildKit secret mounts for build time, Compose secrets and files for run time, and a few rules for rotation. All values in the examples are fake (...EXAMPLE...) and were run on Docker 29 with Compose v5.

What not to do

There are three classic mistakes. Here is one image that makes all of them:

DOCKERFILE
FROM alpine:3
ARG BUILD_TOKEN
ENV API_KEY=sk-live-EXAMPLE-1234
RUN echo "token was $BUILD_TOKEN" > /build.log
CMD ["sleep", "300"]

Build it with a build argument and see what an attacker who can pull the image (or just run docker history) learns.

ENV is public

CONSOLE
$ docker image inspect leaky --format '{{json .Config.Env}}'
["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","API_KEY=sk-live-EXAMPLE-1234"]

Every container started from the image inherits API_KEY, and so does every person who pulls it. An ENV value is stored in the image configuration in plain text.

ARG is not secret either

Build arguments look temporary, but they are recorded in the image history:

CONSOLE
$ docker build -t leaky --build-arg BUILD_TOKEN=tok-EXAMPLE-9999 .
$ docker history --no-trunc leaky --format '{{.CreatedBy}}'
CMD ["sleep" "300"]
RUN |1 BUILD_TOKEN=tok-EXAMPLE-9999 /bin/sh -c echo "token was $BUILD_TOKEN" > /build.log # buildkit
ENV API_KEY=sk-live-EXAMPLE-1234
ARG BUILD_TOKEN=tok-EXAMPLE-9999
CMD ["/bin/sh"]

The token appears twice: in the ARG line and in the RUN line that used it. And because the RUN wrote it into /build.log, it also sits in the filesystem of a layer:

CONSOLE
$ docker run --rm leaky cat /build.log
token was tok-EXAMPLE-9999

Copying secret files and deleting them later

COPY .env . followed by RUN rm .env does not remove anything. Layers are immutable: the file exists in the layer that copied it, and docker save or any registry pull exposes it. Keep secret files out of the build context with a .dockerignore.

-e and docker inspect

Passing a secret with -e at run time keeps it out of the image, but it is still easy to read:

CONSOLE
$ docker run -d --name app -e DB_PASSWORD=hunter2-EXAMPLE leaky
$ docker inspect app --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -E 'API_KEY|DB_PASS'
DB_PASSWORD=hunter2-EXAMPLE
API_KEY=sk-live-EXAMPLE-1234
$ docker exec app sh -c 'cat /proc/1/environ | tr "\0" "\n" | grep DB_PASS'
DB_PASSWORD=hunter2-EXAMPLE

Anyone who can run docker inspect (members of the docker group, monitoring agents) sees it. Inside, any process of the container can read /proc/<pid>/environ, applications often dump their environment into error reports, and child processes inherit it. Environment variables are fine for non-secret settings; for secrets they are the weakest option that still works.

If a secret ever reached an image, a Git repository, a log or a chat, treat it as compromised: rotate it. Deleting the file or rewriting history does not undo the exposure.

Build-time secrets: BuildKit secret mounts

Sometimes the build needs a credential: a token for a private package registry, a license file, a key to download a private dependency. BuildKit (the default builder) solves it with RUN --mount=type=secret. The secret is mounted as a file for that one RUN instruction only; it is never written into a layer or the history.

DOCKERFILE
# syntax=docker/dockerfile:1
FROM alpine:3
RUN --mount=type=secret,id=npm_token \
    sh -c 'echo "using token of length $(wc -c < /run/secrets/npm_token)" > /build.log'
CMD ["cat", "/build.log"]

The id is a label that links the mount to the value you pass on the command line. Pass a file with src=:

CONSOLE
$ echo -n "tok-EXAMPLE-5555" > npm_token.txt
$ docker build --secret id=npm_token,src=npm_token.txt -t clean .
$ docker run --rm clean
using token of length 16

The build used the secret (the token is 16 bytes) and the result only records its length. Now look for traces:

CONSOLE
$ docker history --no-trunc clean --format '{{.CreatedBy}}'
CMD ["cat" "/build.log"]
RUN /bin/sh -c sh -c 'echo "using token of length $(wc -c < /run/secrets/npm_token)" > /build.log' # buildkit
CMD ["/bin/sh"]
$ docker save clean -o clean.tar && mkdir x && tar -xf clean.tar -C x
$ grep -rl "tok-EXAMPLE-5555" x | wc -l
0

The history shows the instruction but not the value, and a text search of every file in the saved image finds zero copies. At run time the file does not exist either:

CONSOLE
$ docker run --rm clean ls /run/secrets
ls: /run/secrets: No such file or directory

You can also pass the secret from an environment variable of your shell (handy in CI), without writing a file:

CONSOLE
$ export NPM_T=tok-EXAMPLE-7777
$ docker build --secret id=npm_token,env=NPM_T -t clean2 .

Three details are worth knowing:

  1. Add required=true (--mount=type=secret,id=npm_token,required=true) when the build must fail without the secret. Without it, a missing secret yields an empty mount and the command inside decides what to do. In my test the build without --secret only printed sh: can't open /run/secrets/npm_token: no such file and carried on.
  2. The cache does not know the value. A step that used a secret can be served from the cache after you change the secret, or even when you pass none at all (my second build without --secret simply reused the cached layer). If the secret must influence the result, pass a non-secret version or --no-cache.
  3. Do not copy the secret into the image yourself. A secret mount protects the file; cp /run/secrets/npm_token /app/ inside the RUN undoes the protection.

For private Git repositories use the SSH variant, RUN --mount=type=ssh, which forwards your SSH agent without ever exposing the key (BuildKit and buildx).

Run-time secrets: files, not variables

For a running container the best portable pattern is: a secret is a file mounted read-only at /run/secrets/<name>, and the application reads the file. Many official images support this with a _FILE variant of their variables, for example POSTGRES_PASSWORD_FILE.

Compose secrets

Compose has a top-level secrets: section. This file starts a database whose password never appears in the environment:

YAML
name: tcd-b10-cs
services:
  db:
    image: postgres:17-alpine
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 3s
      retries: 10

secrets:
  db_password:
    file: ./secrets/db_password.txt
CONSOLE
$ printf 'pg-pass-EXAMPLE-1' > secrets/db_password.txt
$ docker compose up -d --wait
$ docker compose exec db sh -c 'ls -l /run/secrets; cat /run/secrets/db_password; echo; env | grep -i pass'
total 0
-rwxrwxrwx    1 root     root            17 Oct 23 15:12 db_password
pg-pass-EXAMPLE-1
POSTGRES_PASSWORD_FILE=/run/secrets/db_password

The only variable is the path to the secret, not its value. Check what docker inspect can see:

CONSOLE
$ docker inspect tcd-b10-cs-db-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -i pass
POSTGRES_PASSWORD_FILE=/run/secrets/db_password

Only the path again. The secret file is not part of the container configuration.

With plain Docker Compose (no Swarm), a file: secret is a read-only bind mount of your host file. On the Windows machine used for this test the container showed it as -rwxrwxrwx and Compose's per-secret mode, uid and gid options were not applied. Protect the file on the host (owner-only permissions, outside the repository, in .gitignore) rather than relying on those options. Swarm secrets honor them.

A service can see only the secrets it lists. Another service in the same project that does not list db_password has no /run/secrets directory. You can also rename the file inside the container with the long form:

YAML
    secrets:
      - source: db_password
        target: pw

Secrets can also come from an environment variable of the shell that runs Compose, which suits CI systems that already keep secrets in variables:

YAML
services:
  app:
    image: alpine:3
    command: ["sh", "-c", "echo \"secret has $(wc -c < /run/secrets/api_key) bytes\"; env | grep -c API_KEY"]
    secrets:
      - api_key

secrets:
  api_key:
    environment: API_KEY_SOURCE
CONSOLE
$ API_KEY_SOURCE=key-EXAMPLE-1 docker compose up
app-1  | secret has 13 bytes
app-1  | 0

The app read 13 bytes from /run/secrets/api_key, and env | grep -c API_KEY counted zero variables: the value arrived as a file, not in the environment. docker compose config prints only the secret's name and target, never its value.

Plain docker run

Without Compose, bind-mount a file read-only, or pipe the value through stdin:

CONSOLE
$ docker run --rm --mount type=bind,src=$PWD/secrets/db_password.txt,dst=/run/secrets/db,readonly alpine:3 cat /run/secrets/db
pg-pass-EXAMPLE-2
$ echo -n "stdin-EXAMPLE" | docker run -i --rm alpine:3 sh -c 'read -r s; echo "got ${#s} chars"'
got 13 chars

The first form is the file pattern; the second suits one-shot programs that read a secret once. Appending to the mounted file fails because it is read-only (Read-only file system).

Keep the file out of the image and out of Git

  • Add secrets/, .env and *.pem to both .gitignore and .dockerignore.
  • Commit an example (.env.example, secrets/db_password.txt.example) with fake values.
  • On servers, store secrets in a directory only the deploy user can read (chmod 600).

Environment variables: when are they acceptable?

For local development and for non-sensitive configuration, .env files and environment: are perfectly fine (Environment, Variables and Secrets). In production, prefer files. If a framework forces you to use an environment variable, set it from a secret store at deploy time rather than writing it into a Compose file, restrict who can run docker inspect, and make sure the application never prints its environment.

Other secret stores

At some point a file on disk is not enough. Typical next steps:

  • Docker Swarm secrets (docker secret create) store the secret encrypted in the cluster and mount it in memory (tmpfs) on the nodes that run the service. See Docker Swarm Basics.
  • Kubernetes Secrets are similar; enable encryption at rest and restrict access.
  • A secrets manager such as HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager or Azure Key Vault, which also provides auditing, access policies and short-lived credentials. The application (or a sidecar/agent) fetches the secret at start-up using an identity it already has, so nothing sensitive lives in your repository at all.

The principle is the same everywhere: deliver the secret to the process at run time through a channel that is not the image, not the Compose file and not the command line.

Rotation

A secret you can never change is a liability. Plan for rotation from the start:

  1. Read secrets at start-up (or on a signal), not at build time. Then a new value only needs a restart, not a rebuild.
  2. Mounted files change when the host file changes. After I overwrote the secret file on the host, a running container saw the new content immediately (cat /run/secrets/db_password printed the new value), but docker compose up -d reported the container as Running and did not restart it. Your application has to re-read the file, or you restart it yourself with docker compose restart db.
  3. Remember what the value is protecting. Changing the file that holds a database password does not change the password inside the database; PostgreSQL's POSTGRES_PASSWORD_FILE is read only when the data directory is first created. To rotate for real, run ALTER ROLE ... PASSWORD ... in the database and update the file.
  4. Overlap old and new. Safe rotations add the new credential, move all clients to it, then revoke the old one. Supporting two valid credentials for a short time avoids downtime.
  5. Prefer short-lived credentials (tokens that expire in minutes or hours) over long-lived passwords; then rotation happens automatically.

Quick audit

Look for leaks in your own images with commands from this lesson:

Terminal
docker history --no-trunc myapp:1.0          # secrets in ARG / ENV / RUN?
docker image inspect myapp:1.0 --format '{{json .Config.Env}}'
docker save myapp:1.0 -o myapp.tar            # then search the extracted layers

Dedicated scanners such as Trivy can also search images for embedded keys and tokens (see Image Security and Scanning).

Clean up

CONSOLE
$ docker compose down -v
$ docker rmi leaky clean clean2

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A Dockerfile contains ARG TOKEN and RUN ./fetch.sh $TOKEN. Is the token safe once the build finishes?

    docker history --no-trunc shows ARG values and the RUN command line. Use RUN --mount=type=secret for build-time credentials.
  2. 2. What is true about RUN --mount=type=secret,id=npm_token?

    BuildKit mounts the secret temporarily (at /run/secrets/<id> by default). After the instruction it is gone; a search of the saved image finds no copy.
  3. 3. Why is passing a password with -e DB_PASSWORD=... considered weak?

    Anyone allowed to inspect the container, and any process inside it, can read the environment. Mounting a file readable only by the app is safer.
  4. 4. In Compose, POSTGRES_PASSWORD_FILE: /run/secrets/db_password together with a secrets: entry achieves what?

    Only the path shows up in docker inspect. Note that Compose does not encrypt a file: secret; it bind-mounts it read-only, so protect the file on the host.
  5. 5. You overwrite the host file behind a Compose file: secret. What happens to a running container?

    The mount points at the host file, so the content changes immediately. docker compose up -d considered the container up to date and left it running.

Summary

  • ENV, ARG, copied files and -e values are readable from the image history, docker inspect, /proc or the layers. Never put secrets there.
  • Use BuildKit secret mounts (--mount=type=secret with --secret id=...,src= or env=) for build-time credentials; add required=true, and remember the cache ignores the value.
  • At run time, deliver secrets as read-only files under /run/secrets (Compose secrets:, bind mounts, or _FILE variables) rather than as environment variables.
  • A file: secret in plain Compose is a bind mount of the host file, so guard that file with permissions, .gitignore and .dockerignore.
  • Rotate by reading secrets at start-up, changing the value where it really lives (the database, the provider), overlapping old and new, and preferring short-lived credentials.
  • If a secret leaked once, rotate it; deleting it afterwards is not enough.

Next: seccomp, AppArmor and Read-Only Filesystems