Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 45 ยท Practice and Review

Practice Exercises with Solutions

Eight PyTorch exercises with worked solutions: tensor shapes, broadcasting, gradients by hand, module design, loop bugs and conv shapes, with live NumPy checks.

You learn PyTorch by predicting what the code will do and then finding out. This page has eight exercises that cover the whole track: shapes, broadcasting, gradients, modules, training-loop bugs and convolution sizes. Each one has a short task, an editor with starter code, and a hidden solution with an explanation.

PyTorch itself cannot run on this site, so every live editor uses NumPy. That is not a compromise for most of these tasks. Shape rules, broadcasting and the arithmetic behind gradients are identical in NumPy and PyTorch, and the exercises ask you to write the PyTorch version of an idea where the library matters. The solutions that show PyTorch code are read-only and clearly marked.

How to practice

Predict first Write down the shape or number before running.
Run the starter It prints what you should match.
Check numerically A finite-difference check never lies.
Peek last Open the solution after a real attempt.

The track lessons are linked before each group.

1 / 2

Shapes and broadcasting

These practice indexing and reshaping and tensor math and broadcasting.

1. Broadcast shape detective (Beginner)

Broadcasting lines shapes up from the right. Two dimensions are compatible when they are equal or one of them is 1; a missing dimension counts as 1. Write broadcast_shape(a, b) that returns the result shape as a tuple, or raises ValueError when the shapes are incompatible. Do not use np.broadcast_shapes: it is the reference we compare against.

Python Exercise 1: Broadcast shape detective

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution
Python
import numpy as np

def broadcast_shape(a, b):
    result = []
    for i in range(1, max(len(a), len(b)) + 1):
        x = a[-i] if i <= len(a) else 1
        y = b[-i] if i <= len(b) else 1
        if x != y and x != 1 and y != 1:
            raise ValueError(f"cannot broadcast {a} and {b}")
        result.append(max(x, y))
    return tuple(reversed(result))

The loop reads one dimension from the right on each pass, treats a missing dimension as 1, and keeps the larger value of a compatible pair. The last case, (2, 3) against (3, 2), fails because 3 and 2 differ and neither is 1. PyTorch applies exactly the same rule and reports the same kind of error when it fails.

2. From channels-last to channels-first (Beginner)

A batch of images arrives as (N, H, W, C), the layout of most image libraries. nn.Conv2d wants (N, C, H, W). Convert it with a single transposition and confirm that pixel (row 1, col 2) of image 0, channel 1 did not move to a different place. In PyTorch you would write x.permute(0, 3, 1, 2); NumPy calls it transpose.

Python Exercise 2: Channels-last to channels-first

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution
Python
converted = batch.transpose(0, 3, 1, 2)

The argument lists which old axis goes to each new position: new axis 0 is old axis 0 (N), new axis 1 is old axis 3 (C), then old axes 1 and 2 (H and W). The check reads converted[n, c, h, w] against batch[n, h, w, c], so for n=0, c=1, h=1, w=2 the two indexes are [0, 1, 1, 2] and [0, 1, 2, 1]. In PyTorch the result of permute is a view with odd strides. Call .contiguous() before .view(...) on it.

3. Standardize without a loop (Intermediate)

Standardize every column of a matrix so each has mean 0 and standard deviation 1, using broadcasting and no Python loop. Then do a row-wise softmax with the max subtracted for numerical stability. The trick in both tasks is keepdims=True (PyTorch: keepdim=True), which keeps the reduced dimension as size 1 so shapes line up for broadcasting.

Python Exercise 3: Standardize and softmax

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution
Python
def standardize(x):
    return (x - x.mean(axis=0, keepdims=True)) / x.std(axis=0, keepdims=True)

def softmax_rows(x):
    shifted = x - x.max(axis=1, keepdims=True)
    e = np.exp(shifted)
    return e / e.sum(axis=1, keepdims=True)

Subtracting the row maximum does not change the softmax (it cancels in the ratio) but it keeps exp from overflowing: exp(1002) is inf, while exp(0) is 1. In PyTorch you would call torch.softmax(x, dim=1), which does this internally. For column statistics the mean has shape (1, 3) and broadcasts against (200, 3) along the rows.

Gradients by hand

These practice computational graphs and autograd in practice.

4. Derive the gradient of mean squared error (Intermediate)

For a one-feature model pred = w * x + b and loss L = mean((pred - y) ** 2), derive dL/dw and dL/db yourself, then verify them with a central finite difference (L(p + h) - L(p - h)) / (2h). This is what loss.backward() computes for you, and the numerical check is how gradcheck works.

Python Exercise 4: Gradient of mean squared error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution
Python
def analytic_grads(w, b):
    err = w * x + b - y
    dw = np.mean(2 * err * x)
    db = np.mean(2 * err)
    return dw, db

With err = w*x + b - y, the chain rule gives dL/dpred = 2 * err / n. Since d pred / dw = x and d pred / db = 1, the gradients are the mean of 2 * err * x and the mean of 2 * err. Here the loss is 20.875, dw is -16.5 and db is -8.5 (the printed analytic and numeric values agree to four decimals). In PyTorch, w = torch.tensor(0.5, requires_grad=True) followed by loss.backward() leaves the same numbers in w.grad and b.grad.

5. Gradient of softmax cross-entropy (Advanced)

Cross-entropy on logits z with the correct class t is L = -log(softmax(z)[t]). The gradient with respect to the logits has a famously clean form. Work it out (hint: it involves the softmax probabilities and a one-hot vector), implement it, and confirm with finite differences. This is why CrossEntropyLoss wants raw logits instead of probabilities.

Python Exercise 5: Softmax cross-entropy gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution
Python
def grad(v):
    g = softmax(v)
    g[target] -= 1.0
    return g

The gradient is softmax(z) - onehot(t): the predicted probabilities, with 1 subtracted at the correct class. The loss is about 1.74, and the gradient is roughly [0.79, 0.04, -0.83]. Wrong classes are pushed down in proportion to how much probability they stole, and the right class is pushed up. The terms cancel so neatly that fusing softmax and log into one loss is both faster and more stable, which is why nn.CrossEntropyLoss applies log_softmax itself.

Module design

These practice modules and layers and the tiny autograd lesson.

6. Build a Linear layer and a Sequential container (Intermediate)

Write a Linear class with weights W of shape (in, out) and bias b of shape (out,), a Sequential that applies layers in order and collects parameter arrays, and a ReLU with no parameters. Then count the parameters of Linear(4, 8) -> ReLU -> Linear(8, 3). A PyTorch nn.Linear(4, 8) stores its weight transposed as (8, 4), but the count is the same.

Python Exercise 6: A Linear layer and Sequential

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution
Python
class Linear:
    def __init__(self, n_in, n_out, rng):
        self.W = rng.normal(0, np.sqrt(2 / n_in), size=(n_in, n_out))
        self.b = np.zeros(n_out)

    def forward(self, x):
        return x @ self.W + self.b

    def parameters(self):
        return [self.W, self.b]

class ReLU:
    def forward(self, x):
        return np.maximum(x, 0)

    def parameters(self):
        return []

class Sequential:
    def __init__(self, *layers):
        self.layers = layers

    def forward(self, x):
        for layer in self.layers:
            x = layer.forward(x)
        return x

    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

The count is 4*8 + 8 + 8*3 + 3 = 67. The same model in PyTorch is nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 3)), and sum(p.numel() for p in model.parameters()) gives 67. The real nn.Module adds automatic parameter registration (assigning a layer to self.name is enough), state_dict, train()/eval() and device movement; the stand-in only shows the structure.

Training-loop bugs

These practice the complete training loop and debugging.

7. Find the three bugs (Intermediate)

The loop below fits y = 3x + 1 by gradient descent in NumPy. It has three bugs that mirror real PyTorch mistakes, and the loss never gets close to zero. Fix them. Hints: one is the update direction, one is gradient bookkeeping (the NumPy twin of forgetting optimizer.zero_grad()), and one is the learning rate.

Python Exercise 7: Find the three bugs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution
Python
lr = 0.3

for epoch in range(100):
    err = w * x + b - y
    grad_w = np.mean(2 * err * x)      # bug 2: assign, do not accumulate
    grad_b = np.mean(2 * err)
    w -= lr * grad_w                   # bug 1: subtract the gradient
    b -= lr * grad_b

Bug 1: w += lr * grad climbs the loss, because the gradient points uphill. Descent subtracts it. Bug 2: += accumulates gradients across epochs, exactly what PyTorch does to .grad when you skip optimizer.zero_grad(): each step then uses the sum of all past gradients. Bug 3: a learning rate of 5.0 is far too large for this problem and makes the updates overshoot and diverge; 0.3 converges. After the fixes the loop prints w = 3.0, b = 1.0.

The PyTorch version of the same mistakes, read-only:

Python
for epoch in range(100):
    pred = model(x)
    loss = loss_fn(pred, y)
    loss.backward()
    optimizer.step()        # bug: zero_grad() is never called
    # fix: call optimizer.zero_grad() before backward() each iteration

Convolution shapes

This practices convolutions and CNNs.

8. Predict conv output sizes (Intermediate)

For one spatial dimension, out = floor((n + 2*padding - dilation*(kernel - 1) - 1) / stride) + 1. Implement conv_out, then trace a 28 by 28 image through the stack Conv2d(k=3, p=1), MaxPool2d(2), Conv2d(k=3, p=0), MaxPool2d(2) and report the size that Flatten would see with 16 channels.

Python Exercise 8: Predict conv output sizes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Show solution
Python
def conv_out(n, kernel, stride=1, padding=0, dilation=1):
    return (n + 2 * padding - dilation * (kernel - 1) - 1) // stride + 1

The sizes go 28, 14, 12, 6, so Flatten sees 16 * 6 * 6 = 576 features. Compute the size once and use it for the first Linear layer (or use nn.LazyLinear, which infers it on the first batch). Padding of 1 with a 3 by 3 kernel keeps the size; the unpadded second convolution shrinks it by 2.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which pair of shapes can be broadcast together?

    Compare from the right: 1 against 5 and 4 against 1 are both fine, giving (4, 5). The other pairs have a mismatched pair of dimensions where neither is 1.
  2. 2. What is the gradient of softmax cross-entropy with respect to the logits?

    The gradient is the predicted probability vector with 1 subtracted at the target class, which is also why raw logits go into the loss.
  3. 3. A loop never calls zero_grad. What happens to the gradients?

    PyTorch adds into .grad on every backward() call, so without zero_grad each step uses the sum of all earlier gradients.
  4. 4. How do you check an analytic gradient numerically?

    The central finite difference approximates the derivative directly from the function, which is what gradcheck automates.

Summary

  • Broadcasting compares shapes from the right; equal sizes or a size of 1 are compatible.
  • keepdim=True keeps reduced axes so results broadcast back against the original tensor.
  • Gradients can always be verified with central finite differences.
  • The softmax cross-entropy gradient is softmax(z) - onehot(t).
  • A module is parameters plus a forward function; a container just chains them and collects parameters.
  • The classic loop bugs are the wrong update sign, forgotten zero_grad() and a bad learning rate.

Next, test yourself with PyTorch Interview Questions.