Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 40 ยท Neural Networks with Flax

Flax Modules: init and apply

Learn how Flax separates a model from its parameters: define layers with nn.Module, create parameters with init, run them with apply, and inspect the parameter tree.

JAX gives you arrays, grad, jit and vmap. It does not give you layers. You could write a neural network as plain functions that take a dict of parameters, and for small models that is a perfectly good approach. For larger models you want named layers, automatic parameter creation, shape inference and reusable building blocks. Flax is the neural-network library that provides these, built by the JAX team at Google and used widely in research.

This lesson introduces Flax's established API, flax.linen, and the one idea that separates it from PyTorch: a Flax module does not own its parameters. The module describes the computation; the parameters live outside, in an ordinary pytree that you pass in. That is what keeps everything compatible with grad and jit.

Flax needs JAX, and JAX cannot run in the browser, so Flax code appears in plain blocks (real code for your own machine: pip install flax optax). The runnable examples build a miniature of the same design in NumPy: a module class with init and apply where the parameters are an external dict. Flax also has a newer, object-style API called NNX, introduced at the end of the last lesson of this chapter.

The mental model

flowchart LR
    M["Module<br/>(describes the computation)"]
    I["init(key, example_input)"]
    P["params: a pytree<br/>(dict of arrays)"]
    A["apply(params, x)"]
    Y["output"]
    M --> I --> P
    M --> A
    P --> A --> Y

Two separate steps: init creates parameters once from an example input; apply is a pure function of (params, x) that you can differentiate, jit and vmap.

Your first Flax model

Python
import jax, jax.numpy as jnp
import flax.linen as nn

class MLP(nn.Module):
    hidden: int = 16
    out: int = 3

    @nn.compact
    def __call__(self, x):
        x = nn.Dense(self.hidden)(x)
        x = nn.relu(x)
        x = nn.Dense(self.out)(x)
        return x

model = MLP()
x = jnp.ones((5, 4))                        # a batch of 5 inputs with 4 features

variables = model.init(jax.random.key(0), x)   # creates parameters
y = model.apply(variables, x)                  # runs the network
print(y.shape)                                 # (5, 3)

What each piece does:

  • nn.Module is a dataclass. Fields such as hidden and out are hyperparameters, not arrays.
  • @nn.compact lets you create sublayers inline inside __call__, where their input shapes are known.
  • model.init(key, x) runs the module once on an example input to discover shapes, creates each layer's parameters using the key, and returns a dict of variables.
  • model.apply(variables, x) runs the forward pass using the supplied parameters.

Note that Dense does not ask for the input size. Flax infers it from x during init, so Dense(16) applied to 4 features creates a (4, 16) kernel.

What the parameters look like

The variables returned by init is a nested dict, a pytree:

Python
print(jax.tree.map(jnp.shape, variables))
# {'params': {'Dense_0': {'bias': (16,), 'kernel': (4, 16)},
#             'Dense_1': {'bias': (3,),  'kernel': (16, 3)}}}

Layers created inside a compact method get automatic names Dense_0, Dense_1, and so on, in call order. You can choose names yourself with name="hidden" when you want stable, readable keys.

Because variables is a pytree, everything from Pytrees Basics applies: count parameters with tree.leaves, scale them with tree.map, save them to disk, or print their shapes as above.

A runnable miniature

NumPy cannot give you Flax, but it can show the design. The class below has an init that builds a parameter dict from a random generator and an example input, and a pure apply that takes that dict. Note that the module object holds only hyperparameters.

Python init and apply: parameters live outside the module (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output shows a (5, 3) result and a parameter tree with the same shape pattern real Flax prints. The same apply with the same dict and input always returns the same output: it is a pure function.

Why parameters live outside

Because apply is a pure function of (params, x):

  1. grad works directly. jax.grad(loss)(params, x, y) differentiates with respect to the parameter dict, no special API.
  2. jit caches cleanly. The compiled function takes arrays in and out; the module is just closed-over structure.
  3. Parameters are data. Save, load, average (ensembles), interpolate (model soups) or shard them with ordinary tree operations.
  4. Reuse is explicit. Calling apply twice with the same params shares weights; with different params, it does not.
Python The same module with different parameters (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Setup versus compact

@nn.compact is the concise style: layers are defined at the point of use. The alternative is a setup method that creates layers as attributes, which is better when several methods share the same layers (for example an autoencoder with separate encode and decode methods):

Python
class AutoEncoder(nn.Module):
    latent: int

    def setup(self):
        self.enc = nn.Dense(self.latent)
        self.dec = nn.Dense(8)

    def encode(self, x):
        return self.enc(x)

    def __call__(self, x):
        return self.dec(self.enc(x))

variables = AutoEncoder(2).init(jax.random.key(0), jnp.ones((1, 8)))
z = AutoEncoder(2).apply(variables, jnp.ones((1, 8)), method=AutoEncoder.encode)

Use method= in apply to call anything other than __call__.

Common beginner mistakes

  • Calling model(x) directly. A Flax module has no parameters of its own, so you must go through apply(variables, x).
  • Reusing the init key for something else. Treat it like any other PRNG key: use once.
  • Passing only variables["params"] to apply. apply expects the whole variables dict, with the "params" key, so that collections such as batch_stats can live beside it. (Passing {"params": params} is the fix.)
  • Changing the input shape after init. Kernels were built for the feature size seen at init; a different feature size fails at apply. Batch size may change freely.
  • Forgetting @nn.compact when defining layers inline in __call__.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In Flax Linen, where do a model's parameters live?

    Modules describe computation; parameters are external data. That keeps apply a pure function compatible with grad and jit.
  2. 2. What does model.init(key, x) need x for?

    Flax infers shapes such as the Dense kernel's input size from an example input during init.
  3. 3. What is the shape of the kernel of nn.Dense(16) applied to inputs with 4 features?

    Flax stores Dense kernels as (input features, output features) and the bias as (output features,).
  4. 4. How do you call a method other than call on a Flax module?

    Because parameters are external, every call goes through apply, with method= selecting the method.

Summary

  • Flax Linen modules describe computation; parameters are a separate pytree.
  • init(key, example) creates the variables; apply(variables, x) runs the model.
  • nn.Dense and friends infer input shapes during init; names default to Dense_0, Dense_1.
  • @nn.compact defines layers inline; setup suits several methods sharing layers.
  • Pure apply means grad, jit, vmap and tree operations work without extra APIs.

Next: Optax and TrainState: Optimizers as Pure Functions