Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 45 ยท Building Models

Modules, Parameters and Layers

Understand nn.Module, forward, nn.Linear, parameters, named_parameters, state_dict, Sequential, ModuleList and train/eval mode, with a live NumPy layer stack.

In the last lesson one line, nn.Linear(1, 1), gave you a complete model. That line created an nn.Module, the building block of every PyTorch network. A module is a Python object that owns some parameters (the tensors that get trained), possibly some submodules (other modules), and a forward method that says how inputs become outputs. Because modules nest, a transformer with hundreds of millions of parameters is built from the same simple pieces as a one-layer linear model.

Understanding what nn.Module does for you pays off every day: how the optimizer finds the weights, how checkpoints are saved, why model.eval() matters. PyTorch cannot run in this browser, so the PyTorch snippets here are read-only. To let you experiment, the live examples build a small Module system in NumPy that registers parameters and submodules the same way and offers named_parameters, state_dict, Sequential, ModuleList and train/eval. It is a stand-in for the bookkeeping, not for PyTorch's automatic differentiation, which is covered in the Autograd chapter.

What a Module gives you

Parameters Trainable tensors, found automatically.
Submodules Layers nest inside layers.
state_dict Save and load every weight by name.
Modes train() and eval() switch behavior.
1 / 2

Your first custom module

To define a model, subclass nn.Module, create layers in __init__, and describe the computation in forward.

Python
import torch
from torch import nn

class TwoLayerNet(nn.Module):
    def __init__(self, n_in, n_hidden, n_out):
        super().__init__()                      # required: sets up the bookkeeping
        self.hidden = nn.Linear(n_in, n_hidden)
        self.act = nn.ReLU()
        self.out = nn.Linear(n_hidden, n_out)

    def forward(self, x):
        return self.out(self.act(self.hidden(x)))

net = TwoLayerNet(4, 8, 3)
y = net(torch.randn(5, 4))      # call the module, not forward()
print(y.shape)                  # torch.Size([5, 3])

Four rules to remember:

  1. Call super().__init__() first. Without it, assigning a layer fails because the internal registries do not exist yet.
  2. Assign layers as attributes (self.hidden = ...). nn.Module overrides attribute assignment: when the value is a Parameter or another Module, it is recorded in an internal dictionary. That registration is how net.parameters() finds everything later.
  3. Write the computation in forward, but call the module (net(x)), not net.forward(x). __call__ runs hooks and other machinery around forward.
  4. Tensors you do not want trained (a fixed mask, running statistics) are registered with self.register_buffer("name", tensor). Buffers move with the module to the GPU and are saved in checkpoints, but they are not parameters.

What nn.Linear holds

nn.Linear(in_features, out_features) owns a weight of shape (out_features, in_features) and a bias of shape (out_features,), and computes x @ weight.T + bias. The weight layout (outputs first) looks backwards at first but makes each row the weights of one output unit. The initial values are random: PyTorch draws them from a uniform distribution scaled by the layer's size (details in Initialization, Normalization and Dropout).

A parameter is just a tensor with requires_grad=True, wrapped in nn.Parameter so modules recognize it:

Python
layer = nn.Linear(3, 2)
print(layer.weight.shape, layer.bias.shape)    # torch.Size([2, 3]) torch.Size([2])
print(layer.weight.requires_grad)              # True

Seeing parameters: parameters(), named_parameters(), state_dict()

Three methods look inside a module and recurse into submodules.

Python
for name, p in net.named_parameters():
    print(name, tuple(p.shape))
# hidden.weight (8, 4)
# hidden.bias (8,)
# out.weight (3, 8)
# out.bias (3,)

total = sum(p.numel() for p in net.parameters())            # 8*4 + 8 + 3*8 + 3 = 67
trainable = sum(p.numel() for p in net.parameters() if p.requires_grad)
  • parameters() yields the tensors only. This is what you hand to an optimizer: SGD(net.parameters(), lr=0.1).
  • named_parameters() yields (name, tensor) pairs with dotted names that mirror the nesting. Handy for printing, freezing some layers or giving them different learning rates.
  • state_dict() returns an ordered dictionary mapping those names to tensors (and buffers as well). It is what you save with torch.save and restore with net.load_state_dict(...), see Saving and Loading Models.

Counting parameters (numel() per tensor) is a quick sanity check that your architecture is the size you intended.

Containers: Sequential, ModuleList, ModuleDict

For a straight pipeline, nn.Sequential saves you writing a class:

Python
net = nn.Sequential(
    nn.Linear(4, 8),
    nn.ReLU(),
    nn.Linear(8, 3),
)
print(net)

Layers are named by position (0.weight, 2.bias, ...), or by name if you pass an OrderedDict. When the data flow is not a straight line (skip connections, several heads), write a class. If you need many similar layers, store them in an nn.ModuleList:

Python
class Stack(nn.Module):
    def __init__(self, depth, width):
        super().__init__()
        self.layers = nn.ModuleList(nn.Linear(width, width) for _ in range(depth))

    def forward(self, x):
        for layer in self.layers:
            x = torch.relu(layer(x))
        return x

The common mistake is using a plain Python list here. The layers would still run in forward, but net.parameters() could not see them, so the optimizer would never update them and the model would silently fail to learn. nn.ModuleDict is the same idea for dictionaries of named modules, for example one output head per task.

train() and eval()

Some layers behave differently while training and while predicting. Dropout randomly zeroes activations during training and does nothing at prediction time. BatchNorm uses the current batch's statistics while training and stored running statistics at prediction time. A module has a boolean flag, training, which net.train() and net.eval() set recursively on every submodule. Forgetting model.eval() before validation is one of the most common sources of noisy, unreproducible metrics. Note that eval() does not turn off gradient tracking; use torch.no_grad() for that (see Autograd in Practice).

A module system in NumPy

The first live example rebuilds the registration machinery. Module.__setattr__ notices parameters and submodules, named_parameters walks the tree, and Linear, ReLU and Sequential compose just like their PyTorch namesakes. Run it and compare the printed names and counts with the PyTorch comments above.

Python A Module stand-in with named parameters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The final line is the "plain list" bug made visible: the model works as a function, but it reports zero parameters, so an optimizer would have nothing to train.

state_dict, train() and eval() in the stand-in

The second example adds state_dict, load_state_dict, train/eval and a Dropout layer, then demonstrates two behaviors you rely on constantly: a saved state restores a model exactly, and the same input gives a different output in training mode (random dropout) than in evaluation mode (deterministic).

Python state_dict round trip and train vs eval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical tips

  • Print a model (print(net)) to see its structure, and sum numel() over parameters() to count weights.
  • Keep __init__ for creating layers and forward for using them. Avoid creating new layers inside forward: they would be re-initialized on every call and never trained.
  • Name layers for readability; you will read those names in state_dict keys and error messages.
  • Use net.apply(fn) to run a function over every submodule, a common way to set a custom initialization.
  • To freeze part of a model, set p.requires_grad = False for its parameters, a core move in Transfer Learning.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You store three nn.Linear layers in a plain Python list inside your module. What goes wrong?

    Only attributes that are Parameters or Modules (including ModuleList) are registered. Use nn.ModuleList for a list of layers.
  2. 2. For nn.Linear(10, 4), what is the shape of weight?

    The weight has shape (out_features, in_features) = (4, 10), and the layer computes x @ weight.T + bias.
  3. 3. What does model.eval() do?

    eval() only switches the training flag of the module tree. Gradient tracking is controlled separately, for example with torch.no_grad().
  4. 4. What is state_dict() mainly used for?

    state_dict() maps names such as 'hidden.weight' to tensors; torch.save stores it and load_state_dict restores it.
  5. 5. What must the first line of a custom module's init be?

    super().init() creates the registries that store parameters and submodules. Without it, attribute assignment fails.

Summary

  • nn.Module owns parameters, submodules and a forward method; assigning a Parameter or Module as an attribute registers it.
  • parameters() feeds the optimizer, named_parameters() gives dotted names, state_dict() is the unit of saving and loading.
  • nn.Linear(in, out) stores weight (out, in) and bias (out,).
  • Use nn.Sequential for pipelines, nn.ModuleList / nn.ModuleDict for collections; never a plain list or dict.
  • train() and eval() switch layers such as Dropout and BatchNorm; they do not disable gradients.
  • Call the module (net(x)), not forward.

Next: the nonlinear functions and the loss functions you plug into these modules, in Activation Functions and Loss Functions.