Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 37 ยท Operating Microservices

Configuration, Feature Flags and Secrets

Keep microservice configuration out of the code: environment variables, a config service, feature flags with gradual rollout, and safe secret handling.

A microservice runs in many places: your laptop, a test cluster, staging, production, and sometimes several regions. The code is the same everywhere, but the database address, the timeouts and the log level are not. Configuration is everything that changes between those places, and the rule that keeps you sane is simple: never bake it into the code or the image.

With dozens of services, configuration also becomes an operations problem. Who changed the timeout? Can we turn a risky feature off without a deploy? Where does the database password live? This lesson covers three tools that answer those questions: environment-based configuration, feature flags, and secrets management.

Three kinds of settings

Config Ports, URLs, timeouts. Differ per environment.
Flags Switch behavior on or off at runtime.
Secrets Passwords and keys. Never in code or logs.

Same build everywhere, different settings around it.

1 / 2

Twelve-factor configuration

The "twelve-factor app" guidelines say to store configuration in environment variables. They are language-neutral, easy to set in Docker, Compose and Kubernetes, and hard to commit by accident. A service reads them once at startup.

Terminal
docker run -e PORT=8080 -e LOG_LEVEL=warn -e DATABASE_URL="$ORDERS_DB_URL" orders:1.4.2

This command is real, but the site cannot run containers, so it is read-only. The idea that matters is the image tag orders:1.4.2: the same image runs in staging and in production, and only the variables differ. If you rebuilt the image per environment, what you tested would not be what you ship.

Environment variables are always strings, so a service should parse and validate them at startup and fail fast if something is missing. A service that crashes at boot with a clear message is far better than one that starts and fails on the first request at 3 a.m.

Do not scatter process.env.X through your code. Read all settings in one module, validate them there, and pass a typed config object to the rest of the program.

A config loader you can run

The stand-in below is a small loader with a schema. It applies defaults, requires some values, checks numbers and allowed values, collects every problem (not just the first), and masks secret values when printing. Real services do this with a library, but the logic is the same.

JS Validating configuration at startup

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The two valid environments share the same code and get different values. The broken one reports all three problems at once, so the operator fixes them in one go. Note the secret flag: logging the whole config is useful for debugging, but only if secrets are masked.

Where configuration comes from

Environment variables are the baseline, but larger systems add layers.

  • Files mounted into the container. Kubernetes ConfigMap objects can be mounted as files or exposed as variables. Good for non-secret settings that several pods share.
  • A configuration service. Tools such as Consul, etcd, or a cloud parameter store hold settings centrally. Services fetch them at startup, or watch for changes and reload without a restart.
  • A hierarchy of defaults. Defaults in code, overridden by a shared file, overridden by per-environment values, overridden by the environment. The most specific wins.

A central store gives you one place for audit history ("who changed this, and when?") and lets you change a value for twenty services at once. The cost is a new dependency: if the config service is down at startup, services cannot boot. Cache the last known good configuration locally, and keep the set of centrally managed values small.

Changing a value live is risky too. Prefer changes that are applied through the normal deploy path, so they are reviewed and can be rolled back, for everything except the settings you deliberately want to flip fast. Those are feature flags.

Feature flags

A feature flag (or feature toggle) is a condition in the code that decides at runtime whether a behavior is on. The new checkout is deployed but dark, and a flag decides who sees it.

Flags separate two events that used to be one: deploying code and releasing a feature. That has big benefits for microservices.

  • Gradual rollout. Turn the feature on for 1%, then 10%, then everyone, watching the metrics at each step.
  • Kill switch. If the feature misbehaves, turn it off in seconds without redeploying.
  • Targeting. Enable it for staff first, or for one country, or for a beta list.
  • Trunk-based development. Unfinished work can be merged behind a flag instead of living on a long branch.

Percentage rollout has a subtle requirement: a user must get the same answer on every request. Flipping a coin per request would show a user the new checkout, then the old one, then the new one again. The standard solution is to hash the flag name and the user id into a bucket from 0 to 99, and turn the feature on when the bucket is below the rollout percentage. Including the flag name means different flags pick different users, so the same unlucky 1% does not always get every experiment.

JS A feature flag evaluator with percentage rollout

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the output from the top. u-1 is on because of the allow list even though its bucket (91) is above 25. u-2 is denied explicitly. u-3 and u-4 fall back to their bucket (29 and 96), and both are at or above 25, so they are off. An unknown flag is off, which is the safe default.

The rollout lines show the hash spreading users evenly: about 10% of 10,000 users at "10%", and so on. The last line is the important property: raising the rollout from 10% to 30% only adds users, so nobody who already saw the feature loses it.

Flag hygiene

Flags are cheap to add and expensive to forget.

  • Give each flag an owner and an expiry. A release flag should be deleted once the rollout reaches 100%. Old flags multiply the number of code paths you must test.
  • Default to the safe state. If the flag service cannot be reached, fall back to the old, proven behavior (or to a cached value).
  • Evaluate locally. Fetch the flag rules and evaluate them in-process, as above. A network call per check would add latency and a new failure mode (The Fallacies of Distributed Computing).
  • Keep flags out of contracts. Do not let a flag silently change an API response that other services depend on without a plan; see API Versioning and Evolving Contracts.
  • Separate kinds. Release flags are short-lived; operational kill switches may live forever; permission flags are really entitlements and belong in your domain model.

Secrets

A secret is any value that grants access: database passwords, API keys, signing keys, tokens. Configuration is "who am I talking to"; a secret is "proof that I may".

The golden rules:

  1. Never commit secrets to the repository, and never bake them into images. Anyone who can pull the image can read the layers.
  2. Keep them out of logs, error messages and URLs. The secret flag in the loader above is the small version of this habit.
  3. Give each service its own credentials, with only the access it needs. A leaked key from the notifications service must not open the payments database.
  4. Use a secrets manager (HashiCorp Vault, a cloud secrets service, or your orchestrator's secret store, ideally encrypted at rest) so that access is audited and values can be rotated without rebuilding.
  5. Rotate on a schedule and after any suspected leak, and design services so a new value can be picked up without a long outage, for example by accepting two valid keys during a transition.

Here is how a Kubernetes pod reads a secret as an environment variable. It is real configuration, read-only here:

YAML
env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: orders-db
        key: url

A Kubernetes Secret is only base64-encoded by default, not encrypted. Enable encryption at rest and restrict who can read secrets, or use an external secrets manager. The next security lessons go deeper: Authentication and Authorization Between Services and Securing the Edge, Data and Secrets.

What is real and what is the stand-in

The environment-variable commands and the Kubernetes snippet are real and read-only, because the site has no container runtime. The two TryIts are stand-ins that implement the logic (validation, hashing into buckets, allow and deny lists) in a few lines. Production flag services add a dashboard, audit log, streaming updates and SDKs for many languages, but the evaluation idea is the same.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should the same container image run in staging and production?

    Building one artifact and injecting configuration avoids differences that were never tested.
  2. 2. What should a service do when a required setting is missing at startup?

    A clear crash at boot is easy to notice and fix, while a half-working service is hard to diagnose.
  3. 3. Why hash the flag name and user id into a bucket for a percentage rollout?

    A random coin flip per request would flicker. A stable bucket gives consistent behavior, and a higher percentage includes everyone from a lower one.
  4. 4. Which is a good habit with release feature flags?

    Leftover flags add untested code paths. Give each one an owner and remove it when it has done its job.
  5. 5. Which statement about secrets is true?

    Separate, minimal, rotatable credentials limit the damage of a leak, and a manager gives you auditing.

Summary

  • Configuration is whatever differs per environment; keep it out of code and images, and build once.
  • Read environment variables in one place, validate them and fail fast, masking secrets when printing.
  • A central config service gives audit and reach, but adds a dependency; cache the last good values.
  • Feature flags separate deploying from releasing: gradual rollout, kill switches, targeting. Use stable hashing for percentages and delete flags when done.
  • Secrets live in a secrets manager, are per service, minimal, rotated and never logged.

Next: Observability: Structured Logs and Metrics.