Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 40 ยท Authentication and Keys

Key Derivation and Key Management

Build HKDF from HMAC, derive many keys from one secret, design a key hierarchy with envelope encryption and rotation, and learn where secrets should live.

Every cipher and MAC you have met needs a key, and real systems need many of them: one to encrypt, one to authenticate, one per user, one per file, a new one every few months. Nobody wants to generate, transport and remember dozens of independent random keys. The standard solution is to keep a small number of strong secrets and derive every other key from them with a key derivation function (KDF).

This lesson covers two jobs that are easy to confuse. The first is turning one good secret (a shared Diffie-Hellman result, a master key) into several independent keys; that is the job of HKDF, which you will build from HMAC in a dozen lines. Turning a weak secret such as a password into a key is a different problem, covered in password hashing. The second half of the lesson is key management: hierarchies, rotation and where secrets are kept.

Why not just hash the secret?

It is tempting to write key = sha256(secret) and mac_key = sha256(secret + b"mac"). This often works, but it is ad hoc: the constructions are not proven, the inputs are not delimited, and it is easy to produce two keys that accidentally collide in purpose. A proper KDF gives you three properties:

  • Independence. Keys derived for different purposes look unrelated; learning one tells you nothing about another.
  • Uniformity. Even if the input secret is not perfectly uniform (a Diffie-Hellman shared secret is a number with structure, not random bytes), the output is.
  • Flexible length. You ask for exactly as many bytes as you need.

HKDF: extract, then expand

HKDF (RFC 5869) has two stages, both built from HMAC.

  1. Extract concentrates the entropy of the input into one fixed-size pseudorandom key (PRK): PRK = HMAC(salt, input_key_material). The salt is a non-secret value; if you have none, use a string of zero bytes.
  2. Expand stretches the PRK into as many output bytes as you want. It computes blocks T(1) = HMAC(PRK, info || 0x01), T(2) = HMAC(PRK, T(1) || info || 0x02), and so on, then concatenates and truncates them. The info string is a label such as b"file encryption v1" that binds the output to its purpose.
flowchart LR
    S["Salt"] --> E["Extract<br/>HMAC(salt, secret)"]
    IKM["Input secret"] --> E
    E --> PRK["PRK<br/>(uniform key)"]
    PRK --> X["Expand<br/>HMAC blocks + counter"]
    I["info label"] --> X
    X --> OKM["Output keys<br/>(any length)"]

Extract makes a uniform key from the secret; expand turns it into labeled output keys.

Python's standard library has no HKDF function, but hmac and hashlib are all you need. The example below also checks the result against the official test vector from RFC 5869 (test case 1), which is how you should validate any crypto code you write from a specification.

Python HKDF from HMAC, checked against the RFC vector

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the last line prints True, your implementation agrees with the standard, byte for byte.

One secret, many independent keys

The info label is what keeps keys apart. The same input secret with different labels yields unrelated outputs, so the encryption key, the MAC key and the nonce prefix of a protocol can all come from one handshake result. Include a version number in the label so you can change the scheme later without key collisions, and put anything else that identifies the context (a user ID, a protocol name) in it too.

Python Deriving separate keys with labels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Roughly half of the bits differ, as they should for unrelated random-looking keys.

Key hierarchies and envelope encryption

Real systems arrange keys in layers:

  • A root key (or master key) lives in the most protected place available, such as a hardware security module (HSM) or a cloud key management service (KMS). It is rarely used and never leaves that place.
  • Key-encryption keys (KEKs) are wrapped by the root key and protect other keys.
  • Data-encryption keys (DEKs) encrypt the actual data. A fresh DEK per file, per record or per user limits the damage of any single leak.

Storing the data encrypted under a DEK, with the DEK itself stored encrypted under a KEK, is called envelope encryption. Its great advantage is cheap rotation: to move to a new KEK you only re-encrypt the small DEKs, not terabytes of data.

flowchart TD
    R["Root key<br/>(HSM or KMS)"] --> K["Key-encryption key<br/>(KEK v2)"]
    K --> D1["Data key for file A<br/>(stored wrapped)"]
    K --> D2["Data key for file B<br/>(stored wrapped)"]
    D1 --> F1["File A ciphertext"]
    D2 --> F2["File B ciphertext"]

Each file has its own data key; only the data keys are wrapped by the next layer up.

Rotation

Keys should be rotated: replaced on a schedule, and immediately after a suspected leak. Rotation limits how much data one key protects, and it makes sure you can actually do it in an emergency. Good designs store a key ID or version next to every ciphertext, so that after rotation old data can still be read with the old key while new data uses the new one.

Python Envelope encryption with key rotation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where secrets should live

  • Not in source code or Git history. A key committed once must be treated as leaked, even if a later commit removes it.
  • Not in client-side code. Anything shipped to a browser or app can be extracted.
  • In a secret manager, KMS or HSM where possible, with access logs and narrowly scoped permissions. Environment variables injected at deploy time are a simpler second best.
  • Separate per environment. Production keys must differ from staging and development keys.
  • Backed up and recoverable. A key you cannot recover means data you cannot recover, so plan for loss as carefully as for theft.
  • Never in logs. Keys, tokens and passwords should never appear in logs, error messages or analytics.

Finally, remember the rule from the beginning: HKDF is for inputs that already contain real entropy. Feeding a human password to HKDF gives you a fast function an attacker can run billions of times per second; use a slow password hash instead.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What are the two stages of HKDF?

    Extract concentrates the entropy of the input into a uniform pseudorandom key, and expand stretches that key into as many labeled output bytes as needed.
  2. 2. What is the purpose of the info label in HKDF?

    The same input secret with different info strings produces independent keys, which is how one handshake result can supply separate encryption and MAC keys.
  3. 3. A user types a password. Which function should turn it into a key?

    Passwords have little entropy, so the derivation must be deliberately slow to make guessing expensive. HKDF is meant for secrets that are already high-entropy.
  4. 4. What is the main benefit of envelope encryption?

    Each record has its own data key, stored wrapped by a higher-level key. Rotating the higher-level key means re-wrapping those small keys while the bulk data stays untouched.
  5. 5. Why store a key ID or version next to each ciphertext?

    After rotation, a system holds several key versions at once. The stored version tells it which one to use for each record.

Summary

  • A KDF turns one strong secret into any number of independent, uniform keys; do not improvise with plain hashes.
  • HKDF = extract (HMAC(salt, secret)) then expand (counter-chained HMAC blocks with an info label). It is easy to build and to check against RFC 5869 test vectors.
  • Use a distinct, versioned info label for every purpose; use a slow password hash, not HKDF, for human passwords.
  • Layer keys (root, key-encryption, data-encryption) and use envelope encryption so rotation is cheap.
  • Store a key version with every ciphertext, rotate on a schedule and after leaks, and keep secrets out of code, logs and clients.

Next: Public-Key Cryptography: The Big Idea, where two parties can agree on keys without ever having met.