Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 45 ยท Building Models

Linear Regression the PyTorch Way

Fit a line the PyTorch way with data, model, loss, optimizer and the five-line training loop, then run the same loop live in NumPy gradient descent.

Linear regression is the "hello world" of machine learning, and it is the perfect first PyTorch model because every ingredient of a large neural network is already present in miniature. You have data, a model with parameters, a loss that measures how wrong the model is, an optimizer that nudges the parameters, and a loop that repeats until the loss stops improving. Once you can read and write this loop, a deep network is "the same loop with a bigger model".

You will meet the loop twice. First in PyTorch, as read-only code, because PyTorch itself cannot run in the browser. Then in NumPy, where we write out by hand everything PyTorch does behind the scenes, and you can run, change and plot it live. The NumPy version is a stand-in for the mechanics (the same math, the same update rule), not a replacement for PyTorch's tooling.

The five parts of every PyTorch model

Data Inputs X and targets y as tensors.
Model Parameters plus a forward pass.
Loss One number: how wrong are we?
Optimizer Moves parameters downhill.

A loop ties them together.

1 / 2

The problem

We observe an input x (say, hours studied) and an output y (say, a score). We believe they are roughly linearly related with some noise: y โ‰ˆ w * x + b. The weight w is the slope, the bias b is the intercept. Training means finding the w and b that make predictions as close as possible to the targets. "As close as possible" needs a number, so we use the mean squared error:

loss = mean((pred - y)^2)

Squaring makes every error positive and punishes big mistakes more than small ones. The loss is a smooth bowl-shaped surface over (w, b), so repeatedly stepping downhill (gradient descent) reaches the bottom.

The PyTorch version

Here is the complete program. It is read-only, but every line is something you will write over and over.

Python
import torch
from torch import nn

torch.manual_seed(0)

# 1. Data: 100 samples, 1 feature. Shape (N, 1) is what nn.Linear expects.
X = torch.rand(100, 1) * 10
y = 3 * X + 2 + torch.randn(100, 1)       # true line y = 3x + 2 plus noise

# 2. Model: one weight and one bias, initialized randomly
model = nn.Linear(in_features=1, out_features=1)

# 3. Loss function
loss_fn = nn.MSELoss()

# 4. Optimizer: plain stochastic gradient descent over the model's parameters
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# 5. The loop
for epoch in range(500):
    pred = model(X)                 # forward pass
    loss = loss_fn(pred, y)         # scalar tensor
    optimizer.zero_grad()           # clear old gradients
    loss.backward()                 # fill .grad of weight and bias
    optimizer.step()                # w -= lr * w.grad, same for b
    if epoch % 100 == 0:
        print(epoch, loss.item())

print(model.weight.item(), model.bias.item())    # close to 3 and 2

Each piece deserves a closer look.

Shapes. nn.Linear expects a batch of samples with shape (N, features). With one feature that is (100, 1), not (100,). A missing trailing dimension is the classic first bug: pred would have shape (100, 1) while y has shape (100,), and the loss would silently broadcast to a (100, 100) table of differences. Always check pred.shape == y.shape (see Tensor Math and Broadcasting).

The model. nn.Linear(1, 1) owns two parameters, weight of shape (1, 1) and bias of shape (1,), already registered with requires_grad=True. Calling model(X) computes X @ weight.T + bias. The next lesson, Modules, Parameters and Layers, opens this up.

The loss. nn.MSELoss() returns a scalar tensor, the mean of the squared differences. A scalar is what backward() needs. Use loss.item() to get a plain Python number for printing, so you do not keep graphs alive in your logs (see Autograd in Practice).

The optimizer. SGD gets the list of parameters it is responsible for and a learning rate lr, the size of each step. optimizer.step() performs param -= lr * param.grad for every parameter. Fancier optimizers such as Adam arrive in Optimizers.

The order inside the loop. Forward, loss, clear gradients, backward, step. If you forget zero_grad(), gradients accumulate from epoch to epoch and the effective step grows.

Epochs. Here each epoch is one pass over the whole data set in a single batch ("full-batch" gradient descent). With big data you split it into mini-batches and do one step per batch, which is the subject of Datasets and DataLoaders.

The same loop, with the math visible

What does PyTorch compute during loss.backward() for this model? With pred = w * x + b and loss = mean((pred - y)^2), the chain rule gives

  • d loss / d w = mean(2 * (pred - y) * x)
  • d loss / d b = mean(2 * (pred - y))

The NumPy version below writes these two lines out by hand. Every comment names the PyTorch line it replaces. Run it, then try changing lr or the number of epochs.

Python Gradient descent for a line, the PyTorch loop in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two things to notice in the printed numbers. The loss starts large and drops fast at first, then slowly (the bowl gets flat near the bottom). And the weight is learned faster than the bias: x ranges up to 10, so changing w moves the predictions much more than changing b, which is why the gradient with respect to w is much larger. Real models have thousands of parameters with differently scaled gradients, one of the reasons adaptive optimizers exist.

Finally, the final w and b are close to, but not exactly, the closed-form least-squares solution. Gradient descent never arrives exactly; it gets closer and closer. For a straight line the closed form exists, so you would never train it with an optimizer in practice. We do it here because the same loop works unchanged when no closed form exists.

The learning rate

The learning rate is the first hyperparameter you tune. Too small and training crawls; just right and it converges quickly; too large and each step overshoots the bottom of the bowl by so much that the loss grows, round after round, until it overflows. The editor below runs the same loop with four rates and plots the loss on a log axis. For this data the threshold for stability is a learning rate of about 0.03; above it the loss explodes.

Python Four learning rates, one loss curve each

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After 60 epochs the smallest rate is still far above the noise floor (loss about 4.9), the moderate ones have converged (about 1.1 to 1.4), and the largest one makes the loss grow every step until it reaches astronomically large values. If you ever see a loss that grows or turns into nan, lower the learning rate first. Learning Rate Schedules covers how to change it during training.

From a line to a network

Nothing in the PyTorch loop is specific to a straight line. To fit a curve, replace the model by something with more parameters and nonlinearities:

Python
model = nn.Sequential(
    nn.Linear(1, 32),
    nn.ReLU(),
    nn.Linear(32, 1),
)

The data, the loss, the optimizer and the five lines of the loop stay exactly the same. That is why PyTorch code looks so uniform from project to project. The remaining chapters fill in the parts: what modules are, which losses and activations to choose, how to feed data in batches and how to evaluate honestly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the correct order of the calls inside a standard PyTorch training step?

    You need the forward pass and loss before backward can run, gradients must be cleared before backward fills them, and step uses the fresh gradients.
  2. 2. pred has shape (100, 1) and y has shape (100,). What happens with loss_fn(pred, y) written as a plain mean((pred - y)**2)?

    Broadcasting turns (100, 1) minus (100,) into a (100, 100) table, and the mean of that is not the intended loss. Reshape y to (100, 1) or squeeze pred.
  3. 3. In gradient descent for a line with MSE, what does one update look like?

    Each parameter moves a small step (lr) against its gradient, which is the downhill direction of the loss.
  4. 4. Your loss grows every epoch and finally becomes nan. What is the most likely first fix?

    Steps that are too large overshoot the minimum more each time, so the loss explodes. Reduce lr (and check for bad data).

Summary

  • Every PyTorch project has data, a model, a loss, an optimizer and a loop; linear regression shows all five with nn.Linear, nn.MSELoss and SGD.
  • A training step is: forward, loss, zero_grad(), backward(), step().
  • Keep shapes explicit: (N, 1) inputs and targets; check pred.shape == y.shape.
  • For y = w*x + b with MSE the gradients are mean(2*(pred - y)*x) and mean(2*(pred - y)), which is what backward() computes for you.
  • The learning rate decides everything: too small is slow, too large diverges.
  • The same loop trains deep networks; only the model changes.

Next, look inside the model: Modules, Parameters and Layers.