Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 40 ยท Random Numbers and Pytrees

Random Sampling Patterns: Distributions, Shuffles and Dropout

Use jax.random distributions, shuffle data with permutation, draw weighted choices, build dropout masks and run Monte Carlo batches with one key per run.

With the key rules from PRNG Keys and split in hand, the rest of jax.random is a familiar menu: distributions, shuffling, weighted choices, masks. This lesson walks through the functions you will use every day, then combines them into three patterns: shuffling a dataset every epoch, dropout, and a Monte Carlo estimate where each run has its own key.

Plain blocks are real JAX (the exact random values depend on your JAX version and configuration, so they are not printed here). Runnable blocks use NumPy generators seeded explicitly, which follow the same one-source-per-purpose discipline. Values will not match JAX.

The signature pattern

Every sampling function looks like jax.random.<dist>(key, shape, dtype=...): the key comes first, then the shape. Parameters such as mean and standard deviation are applied by hand afterwards, because the sampler draws from the standard distribution.

Python
import jax, jax.numpy as jnp

key = jax.random.key(0)
k1, k2, k3, k4 = jax.random.split(key, 4)

z = jax.random.normal(k1, (1000,))               # standard normal
x = 5.0 + 2.0 * z                                # mean 5, std 2
u = jax.random.uniform(k2, (3, 4), minval=-1.0, maxval=1.0)
b = jax.random.bernoulli(k3, p=0.3, shape=(10,)) # booleans, True with prob 0.3
i = jax.random.randint(k4, (5,), 0, 10)          # integers in [0, 10)

Other distributions follow the same shape: exponential, gamma, beta, poisson, laplace, truncated_normal, multivariate_normal, dirichlet, t, gumbel, cauchy. Check the signature in your version for the exact parameter names.

Python Scale and shift a standard normal (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sample mean and standard deviation land close to 5 and 2: shifting and scaling z is all normal(loc, scale) does.

Weighted and unweighted choices

jax.random.choice(key, a, shape, replace, p) picks items from an array or from range(a). For picking one category with given probabilities, jax.random.categorical(key, logits) samples directly from log-probabilities, the natural form for classifiers.

Python
items = jnp.array([10, 20, 30, 40])
pick = jax.random.choice(key, items, shape=(5,), replace=True,
                         p=jnp.array([0.1, 0.1, 0.1, 0.7]))

logits = jnp.log(jnp.array([0.1, 0.1, 0.1, 0.7]))
cat = jax.random.categorical(key, logits, shape=(5,))

Sampling without replacement (replace=False) uses a permutation internally, so the size you ask for cannot exceed the population.

Shuffling: permutation, not in-place

JAX arrays are immutable, so there is no shuffle in place. jax.random.permutation(key, x) returns a shuffled copy of x (or a shuffled arange(x) when x is an integer). Shuffle data and labels with the same permutation to keep pairs aligned.

Python
def shuffle_pair(key, X, y):
    perm = jax.random.permutation(key, X.shape[0])
    return X[perm], y[perm]

For every epoch, derive a new key: epoch_key = jax.random.fold_in(base_key, epoch).

Python One permutation keeps rows and labels aligned (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each epoch (seed) gives a different order, and the pairing never breaks, because rows and labels are reordered by one shared permutation.

Dropout: a random mask times the input

Dropout zeroes a random fraction of activations during training and rescales the rest so the expected value is unchanged. In JAX it is a Bernoulli mask:

Python
def dropout(key, x, rate, training):
    if not training or rate == 0.0:
        return x
    keep = 1.0 - rate
    mask = jax.random.bernoulli(key, keep, x.shape)
    return jnp.where(mask, x / keep, 0.0)

Here training is a Python bool decided before tracing (a static value), so the Python if is fine. The key must be a fresh one for each layer and each step: derive it with fold_in(base_key, step) and split per layer.

Python Dropout preserves the expected value (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About 40% of values become zero, the survivors are scaled up to 1/0.6, and the mean stays close to 1. This inverted dropout scaling means nothing needs to change at test time except switching the mask off.

Monte Carlo with one key per run

To estimate something by simulation, write the single-run function, then vmap it over an array of keys. Each run uses its own key, so the runs are independent, and the whole batch compiles into one program.

Python
def one_run(key):
    xy = jax.random.uniform(key, (2,))               # a random point in the unit square
    return (xy[0] ** 2 + xy[1] ** 2) < 1.0           # inside the quarter circle?

keys = jax.random.split(jax.random.key(0), 100_000)
inside = jax.vmap(one_run)(keys)
print(4.0 * inside.mean())                           # approximately 3.14

In a real run you would wrap the last two lines in jax.jit and get the answer in milliseconds, even on a CPU.

Python Estimating pi: independent runs (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four independent streams give four slightly different estimates around 3.14, and their average is steadier than any one. Averaging correlated streams (the same key reused) would not tighten the estimate at all.

A checklist for random code

  1. Create one root key at the top of the program.
  2. split before every use, or fold_in an index.
  3. Pass keys as function arguments; never hide them in globals.
  4. Under vmap, map over an array of keys.
  5. Draw the data shape first, scale and shift afterwards.
  6. Write tests that fix the key, so failures reproduce.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you draw samples with mean 5 and standard deviation 2 using jax.random.normal?

    The sampler draws the standard normal; you scale and shift the result yourself.
  2. 2. Why does JAX offer permutation instead of an in-place shuffle?

    Immutable arrays cannot be modified in place; permutation returns a shuffled copy or an index permutation.
  3. 3. How do you run 1000 independent Monte Carlo simulations with vmap?

    Each run needs its own key so the runs are independent; vmap maps the single-run function over the key array.
  4. 4. What does inverted dropout do to surviving activations?

    Dividing by the keep probability keeps the expected activation the same, so inference needs no rescaling.

Summary

  • Samplers take (key, shape) and draw standard distributions; scale and shift yourself.
  • choice, categorical and bernoulli cover weighted picks and masks; permutation shuffles immutably.
  • Dropout is a Bernoulli mask with x / keep, with a fresh key each layer and step.
  • Monte Carlo: split one key per run and vmap the single-run function.
  • One root key, split before every use, keys as arguments.

Next: Pytrees Basics: Nested Data JAX Understands