Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 40 ยท Random Numbers and Pytrees

PRNG Keys and split: Random Numbers the JAX Way

Learn why JAX uses explicit PRNG keys instead of a global seed, how jax.random.split and fold_in work, and the one rule that prevents correlated random numbers.

NumPy's random numbers hide a global state. Every call to np.random.rand() quietly changes a hidden generator, so the next call gives something different. That is convenient, but it clashes with how JAX works. JAX functions must be pure: the same inputs always give the same outputs, with no hidden state. Pure functions can be compiled, cached, run in parallel and differentiated safely.

So JAX does something different. There is no global generator. Randomness comes from an explicit key that you pass to every random function, and you create new keys by splitting. Once this clicks (it takes about ten minutes), the rest of JAX's random API is simple.

Plain code blocks are real JAX. The runnable examples use NumPy's SeedSequence, which has the same splitting idea (it is the closest NumPy analogue), so the numbers will not match JAX's. They demonstrate the rules, not the values.

Keys are values, not state

A key is a small piece of data (two unsigned 32-bit integers for legacy keys, an opaque typed scalar for new-style keys) that fully determines a stream of random numbers.

Python
import jax

key = jax.random.key(42)          # new-style typed key
# older code: key = jax.random.PRNGKey(42)   (a uint32 array of shape (2,))

x = jax.random.normal(key, (3,))
y = jax.random.normal(key, (3,))
print(bool((x == y).all()))       # True: same key, same numbers

Calling normal twice with the same key returns the same numbers. That is not a bug. A function of (key, shape) is pure, so it must always return the same result.

The golden rule: never reuse a key

Reusing a key for two independent purposes produces correlated (here, identical) randomness, which silently breaks things like weight initialization and dropout masks. The rule is simple:

Use each key once. To get more randomness, split the key.

Python
key = jax.random.key(0)
key, subkey = jax.random.split(key)      # two new keys, the old one is now spent
a = jax.random.normal(subkey, (3,))

key, subkey = jax.random.split(key)
b = jax.random.normal(subkey, (3,))      # independent of a

jax.random.split(key) returns two new keys; split(key, num=5) returns five. The conventional pattern keeps one key for "the rest of the program" and hands a fresh subkey to each consumer.

flowchart TB
    K0["key (seed 0)"] --> S1{"split"}
    S1 --> K1["key"]
    S1 --> U1["subkey 1 -> use once"]
    K1 --> S2{"split"}
    S2 --> K2["key"]
    S2 --> U2["subkey 2 -> use once"]
    K2 --> S3{"split"}
    S3 --> K3["key ..."]
    S3 --> U3["subkey 3 -> use once"]

A key is a tree you grow. Consume the leaves; keep splitting the trunk.

A runnable analogue

NumPy's SeedSequence.spawn plays the role of split: each spawned child seeds an independent stream. The same-seed-same-numbers rule is identical.

Python Same key, same numbers; split keys, new numbers (NumPy analogue)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The correlation between two long draws from different children is close to zero, as you want. Doing normal(key, 1000) twice and correlating would give exactly 1.0.

Why explicit keys? Three payoffs

  1. Reproducibility. The same key always gives the same result, even on a different machine or number of devices. No hidden counter depends on call order.
  2. Parallel and compiled code. A pure function can be jitted, vmapped or run on eight GPUs without worrying about who touches the global state first.
  3. No accidental coupling. Adding a new random call somewhere never changes the numbers drawn elsewhere, because each part owns its own key.

split many keys at once

Splitting into an array of keys is cheap, and works naturally with vmap: one key per example, per layer, or per step.

Python
keys = jax.random.split(jax.random.key(0), 4)       # an array of 4 keys
samples = jax.vmap(lambda k: jax.random.normal(k, ()))(keys)
print(samples.shape)                                 # (4,)

Each of the four samples uses its own key, so they are independent. Using a single key inside the mapped function would give four identical numbers, the most common randomness bug in vmapped code.

Python One key per example (NumPy analogue)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

fold_in: derive a key from a number

jax.random.fold_in(key, data) mixes an integer into a key and returns a new key. It is the right tool when the index is known and you want a deterministic key per step or per device, without carrying split state around.

Python
base = jax.random.key(0)
step_key = jax.random.fold_in(base, 1000)       # key for training step 1000
device_key = jax.random.fold_in(step_key, 3)    # then specialize for device 3

Use fold_in for "this key belongs to step i", and split for "I need several keys right now". For a training loop, deriving fold_in(base, step) makes the run restartable: resuming from step 5000 reproduces exactly the randomness the original run saw, with no saved generator state.

Keys and jit

A key is an ordinary array argument, so pass it into jitted functions rather than creating it inside with a fixed seed.

Python
@jax.jit
def noisy(key, x):
    return x + 0.1 * jax.random.normal(key, x.shape)

If you hard-code jax.random.key(0) inside the function, every call draws the same noise, because the key becomes a compile-time constant. Pass the key as an argument and split outside.

Key types: new-style versus legacy

  • jax.random.key(seed) creates a typed key. It carries its generator implementation, rejects arithmetic by mistake, and is the recommended form in new code.
  • jax.random.PRNGKey(seed) creates the older raw uint32[2] array. It still works everywhere.
  • The default algorithm is Threefry, a counter-based generator that is reproducible across devices. Other implementations (such as rbg) are faster on accelerators but give different numbers, and can be selected through configuration.

Common mistakes

  • Reusing a key for dropout and initialization.
  • Splitting and then using the parent key again as well as a child.
  • Calling jax.random.key(0) inside a loop and expecting different numbers.
  • Forgetting a per-example key under vmap (all examples get the same noise).
  • Comparing random results across libraries: JAX, NumPy and PyTorch give different numbers for the same seed.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens if you call jax.random.normal(key, (3,)) twice with the same key?

    Random functions are pure functions of the key. There is no hidden state, so the same key always yields the same numbers.
  2. 2. What is the correct way to get more random numbers after using a key?

    split returns fresh, independent keys. Each key should be used for exactly one random call.
  3. 3. When is jax.random.fold_in most useful?

    fold_in(key, i) gives a key specific to index i without storing split state, which makes runs restartable.
  4. 4. A vmapped function draws noise using one shared key. What is the result?

    Same key means same numbers. Split a key per example and map over the array of keys.

Summary

  • JAX has no global random state: every random function takes an explicit key and is pure.
  • Same key gives the same numbers; use each key once and split to get more.
  • Split one key per example, layer or step; use fold_in to derive keys from indexes.
  • Pass keys into jitted functions as arguments instead of fixing a seed inside them.
  • Prefer jax.random.key in new code; values differ from NumPy and other libraries.

Next: Random Sampling Patterns