Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 37 ยท Security

Securing the Edge, Data and Secrets

Harden a microservice system: input validation at the gateway, least privilege, secret rotation, audit logs and PII handling, with runnable stand-ins.

The previous lesson was about proving who is calling. This one is about everything else that keeps a distributed system safe: what you let in at the edge, what each service is allowed to touch, how secrets are rotated, who can see personal data, and how you find out afterward what happened.

Microservices change the security picture in two opposite ways. Each service is smaller, so a breach can be contained better. But there are many more network paths, credentials and data copies, so the attack surface is larger. The goal is defense in depth: several independent layers, so that when one fails, the next one still holds.

Layers of defense

Edge Validate and limit what comes in.
Service Least privilege for each service.
Data Minimize, mask and encrypt personal data.
Audit Record who did what.

Assume any single layer can fail.

1 / 2

Validate input at the edge

The API gateway (API Gateway and Backend for Frontend) is the first place to reject bad requests. Cheap checks there protect every service behind it:

  • Size limits on bodies and headers, so a huge payload never reaches your code.
  • Syntax and schema validation: valid JSON, correct types, required fields present, values in range, and no unknown fields. Rejecting unexpected fields stops "mass assignment" tricks, such as an extra "isAdmin": true.
  • Rate limits per client (Rate Limiting, Load Shedding and Backpressure).
  • Authentication of the caller, as covered earlier.

Validate with an allow list (describe exactly what is acceptable), never a deny list (try to describe everything bad). Note, though, that edge validation is a first line, not the only one. Each service must still validate what it receives, because it can also be called from inside, or from a service that was compromised, or by a message from a broker. And validation is no replacement for safe handling: always use parameterized queries and output encoding, never string concatenation, to prevent injection.

Here is a typical body-size limit and the basic edge settings on an NGINX gateway. This is real configuration, read-only here:

NGINX
server {
  listen 443 ssl;
  client_max_body_size 1m;          # reject larger bodies with 413
  client_body_timeout 10s;          # slow-sending clients are cut off
  add_header X-Content-Type-Options nosniff always;

  location /orders {
    limit_req zone=per_ip burst=20 nodelay;
    proxy_pass http://orders;
  }
}

A validating gateway

The stand-in below is a small gateway that checks the size, parses the JSON, validates it against a schema, and either forwards the request or answers with the right status: 413 too large, 400 malformed, 422 well-formed but invalid. The schema accepts only known fields, checks types and formats with allow-list patterns, and collects all errors so a client can fix them in one round.

Node.js Request validation at the gateway

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Only the first request reaches the backend. The suspicious one produced four errors at once, including the extra isAdmin field and the injection-looking SKU that fails the allow-list pattern. Remember that the pattern check is defense in depth: the order service must still use parameterized queries, and never rely on the gateway having cleaned the data. The status codes also matter to clients: a 4xx tells them to change the request, so they must not retry it unchanged (Timeouts, Retries and Exponential Backoff).

Least privilege

Least privilege means every identity gets exactly the access it needs and no more. In microservices there are several identities to restrict.

  • Database accounts. Each service has its own database user, limited to its own schema (Database per Service and Data Ownership). A reporting service gets read-only access. Nobody uses the admin account at runtime.
  • Network paths. Only allowed pairs of services can talk. In Kubernetes, a NetworkPolicy can say that the database accepts connections only from the orders pods.
  • Cloud permissions. Each service role may touch only its own queue, bucket or secret.
  • Container settings. Run as a non-root user, with a read-only filesystem where possible, and without extra capabilities.
  • Tokens. Narrow audience and scope per call.

Two real, read-only examples: a database user that can only read and write the tables of one service, and a network rule restricting who may reach the orders database.

SQL
CREATE ROLE orders_service LOGIN PASSWORD 'from-the-secrets-manager';
GRANT USAGE ON SCHEMA orders TO orders_service;
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA orders TO orders_service;
-- No DELETE, no DDL, and no access to other schemas
YAML
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: orders-db-from-orders-only
spec:
  podSelector:
    matchLabels: { app: orders-db }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: orders }
      ports:
        - port: 5432

(The SQL uses PostgreSQL syntax; a NetworkPolicy only takes effect when your cluster's network plugin enforces it.)

Secrets and rotation

Configuration, Feature Flags and Secrets covered where secrets live. Here is how they change. Secrets should be rotated: replaced on a schedule (and immediately after a suspected leak), so that a stolen secret stops working. The difficulty is doing it without downtime, because many services and a running system depend on the old value.

The standard answer is a key ring with identifiers. Each signed message or token carries a key id (kid). A rotation then goes in four steps:

  1. Add the new key to every verifier, while signers still use the old one.
  2. Switch signers to the new key. New messages use it; old messages remain valid.
  3. Wait until old messages and tokens have expired.
  4. Retire the old key.

At no step does a valid message get rejected. The simulation below does exactly that with HMAC-signed messages (for example webhooks) and prints the result at each stage.

Node.js Rotating a signing key without downtime

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Through steps 2 and 3, both messages verify. Only after the old key is retired in step 4 is the old message rejected, which is the intended result: if it were still valid and unexpired, you would have retired the key too early. Database passwords rotate similarly: create a second credential, move services over, then delete the first. Dynamic secrets (as offered by some secrets managers) go further and issue each service a unique, short-lived database credential on demand, so there is nothing long-lived to steal.

Personal data across services

Microservices spread data. A customer's name might be in orders, shipping, billing, analytics and a search index, plus in logs and backups. Each copy is a liability under privacy rules such as GDPR, which includes the right to erase data.

  • Minimize. Give each service only the fields it needs. Payments needs the card token, not the home address; marketing needs the email, not the card.
  • Reference by id. Store a customer id and fetch details from the owning service when needed, instead of copying them everywhere (Database per Service and Data Ownership).
  • Tokenize or pseudonymize. Replace values with a keyed hash or a token, so analytics can count unique customers without seeing emails. Anything keyed can be re-linked by whoever holds the key, so protect it.
  • Mask in logs and traces. Never write full emails, card numbers or tokens to logs (Observability: Structured Logs and Metrics).
  • Encrypt in transit (TLS/mTLS) and at rest, and give sensitive columns extra protection.
  • Plan deletion. Events and backups make erasure hard. Prefer storing personal data in one owning service, with other services holding only ids, so erasing is one operation, and consider encrypting per-user data with a per-user key you can destroy ("crypto-shredding").

Never use real card numbers in tests: the number below is a widely published test card.

Audit logs

When something goes wrong, you need to answer "who did what, when, and to what?" An audit log is a separate, append-only record of security-relevant actions: logins, permission changes, data exports, refunds, administrative access to personal data. Make it different from ordinary application logs: record the actor, the action, the target and the time, not the sensitive data itself; store it where service operators cannot silently edit it; and keep it for the retention period your rules require.

To make tampering evident, each entry can include a hash of the previous one, forming a chain: change any old entry and every later hash stops matching. Production systems also write to write-once storage or an external system. The stand-in below combines this with the data-minimization ideas: it shows what each service sees of one customer, how to mask and pseudonymize values, and a hash-chained audit log that detects an edited entry.

Node.js Data minimization, masking and a tamper-evident audit log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Each service gets a different slice of the same customer, and the masked or pseudonymized values are what belong in logs and analytics. The audit chain verifies as intact, until one old entry is edited, at which point verification pinpoints entry 2. A real system would also protect the chain's latest hash outside the attacker's reach, since someone who can rewrite every entry could recompute all the hashes.

What is real and what is the stand-in

The practices here are standard: gateway validation, least-privilege roles and network policies, key rotation with key ids, data minimization, masking and audit trails. The NGINX, SQL and Kubernetes snippets are real and read-only. The TryIts are small, in-memory stand-ins: the schema checker is hand-written (real gateways use JSON Schema or OpenAPI validation), the key ring has no secrets manager behind it, and the audit log is an array, not write-once storage. Security is a field of its own; treat this lesson as the map of what to study next, not as a complete program, and consult the OWASP guidance for the details of each attack class.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should a gateway reject unknown fields in a request body?

    An allow-list schema accepts only what the endpoint expects, so smuggled fields never reach the backend.
  2. 2. Does validating at the gateway mean services can skip their own validation?

    Defense in depth: other callers, messages or a compromised service can bypass the gateway.
  3. 3. What is the safe order of a key rotation?

    Verifiers learn the new key first, so nothing valid is rejected at any point. The old key goes last.
  4. 4. Which is an example of least privilege?

    Each identity gets only the access it needs, which limits what a compromise can reach.
  5. 5. What makes a hash-chained audit log tamper-evident?

    Verification recomputes the chain and reports where it first breaks. Keep the latest hash somewhere an attacker cannot rewrite.

Summary

  • Layer defenses: validate at the edge, restrict each service, minimize and protect data, and audit actions.
  • Gateway validation uses size limits, allow-list schemas and rejection of unknown fields, but services must still validate and use parameterized queries.
  • Least privilege applies to database users, network paths, cloud roles, containers and tokens.
  • Rotate secrets with key ids so verifiers accept both keys during the transition, then retire the old one.
  • Minimize personal data across services, mask it in logs, and keep a separate, append-only, tamper-evident audit log.

Next: Testing Microservices.