Skip to content
Tutorials Center
My learning

Topics

Lesson 71 of 73 ยท Practice and Review

Practice Exercises

Ten graded deep learning exercises with runnable solutions: neurons, softmax, backpropagation, optimizers, dropout, convolutions, attention, layer norm and parity.

You learn deep learning by writing it, not by reading it. This page collects ten exercises that revisit the whole tutorial, from a single neuron to attention, ordered from warm-ups to a small challenge. Every solution is a runnable example that checks itself with assert statements, so a silent run means the answer is right and an error tells you what is wrong.

How to use the page. Read an exercise, then try it before looking at the solution: write your own code in a blank editor, a notebook or on paper. Only then run the solution block and compare. If your answer disagrees, the asserts and the printed numbers help you find out where. The solutions deliberately use the plain NumPy style of the rest of the tutorial.

Four levels

Warm-up Neurons, softmax, counting parameters.
Core Backprop check, optimizers, dropout.
Advanced Convolution, attention, layer norm.
Challenge Teach a network 3-bit parity.

Attempt first, then compare with the solution.

1 / 2

Level 1: Warm-up

Exercise 1: a NAND neuron. Choose weights and a bias for a single artificial neuron with a step activation (output 1 if the weighted sum is above 0) so that it computes NAND: the output is 0 only when both inputs are 1. Hint: AND needs both inputs on to cross the threshold; NAND flips the signs.

Exercise 2: stable softmax and cross-entropy. Write softmax(z) and cross_entropy(logits, labels) that do not overflow for logits near 1000. Verify that a model that outputs ten equal logits has a loss of exactly log(10). Hint: subtract the row maximum first; work with log-probabilities. See Loss Functions.

Exercise 3: shapes and parameters. For an MLP with layer sizes 784, 128, 64 and 10, compute the number of trainable parameters by formula, and show the shape of the activations when a batch of 32 flows through it. Hint: a layer from a to b units has a * b + b parameters. See Layers and the Forward Pass.

Python Solutions to exercises 1 to 3

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The NAND weights are -1, -1 with a bias of 1.5: only the input (1, 1) pushes the sum below zero. The softmax of [1000, 1001, 999] is the same as the softmax of [0, 1, -1], because softmax ignores a constant shift, which is exactly what makes the stable version work. The network has 109,386 parameters, most of them in the first layer.

Level 2: Core techniques

Exercise 4: check a backward pass. For a network tanh(X @ W1 + b1) @ W2 + b2 with a mean-squared-error loss, write the gradients for all four parameters by hand, then verify each against central finite differences. The relative error should be below 1e-6. Hint: the derivative of tanh is 1 - tanh(z)**2. See Backpropagation by Hand and Gradient Checking.

Exercise 5: optimizers on a valley. Minimize f(x, y) = 0.05 x^2 + 5 y^2 from the point (5, 1) for 100 steps with plain gradient descent (learning rate 0.15), with momentum (rate 0.05, beta 0.9) and with Adam (rate 0.3). Which gets closest to zero, and why does plain descent move slowly? Hint: the valley is 100 times steeper in y than in x, so the learning rate is limited by y. See Momentum, RMSProp and Adam.

Exercise 6: inverted dropout. Implement dropout(x, p, rng, train) so that the expected value of the output equals the input during training, and so that nothing changes in evaluation mode. Verify with 100,000 ones. See Dropout and Data Augmentation.

Python Solutions to exercises 4 to 6

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The gradient errors are around 1e-10, so the hand-derived backward pass is right. In the valley, plain descent can only use a step small enough for the steep y direction, so it crawls along x and ends at about 0.06, while momentum and Adam finish near 5e-5. Dropout zeroes about 30 percent of the entries, and the survivors are scaled by 1 / 0.7 so that the average stays at 1: that is why inverted dropout needs no change at test time.

Level 3: Convolutions, attention and normalization

Exercise 7: convolution by hand. Write out_size(n, k, pad, stride). Implement a "valid" 2D convolution (really cross-correlation) of an 8 by 8 image with a 3 by 3 kernel using loops, and check it against scipy.signal.correlate2d. Compute the output width for a 28-pixel input with a 5 by 5 kernel, padding 2 and stride 2, and the parameter count of a conv layer with 3 input channels, 16 filters and 3 by 3 kernels. See The Convolution Operation and Convolution, Pooling and Feature Maps.

Exercise 8: causal attention. Implement scaled dot-product attention with an optional causal mask, so that position i can only look at positions up to i. Assert that each row of weights sums to 1, that the weights above the diagonal are zero, and that the first output row equals the first value vector. See The Attention Mechanism.

Exercise 9: layer normalization. Implement the forward pass of layer norm (normalize each row, then scale and shift) and its backward pass for the input. Verify the backward pass with finite differences. Hint: the compact formula is dx = (dxhat - mean(dxhat) - xhat * mean(dxhat * xhat)) / sqrt(var + eps), where dxhat = dy * gain. See Batch Norm and Layer Norm.

Python Solutions to exercises 7 to 9

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The attention printout is a lower-triangular matrix: row 0 puts all its weight on itself, row 1 splits between positions 0 and 1, and so on. That triangular shape is the causal mask, and it is why a language model can be trained on a whole sentence at once without peeking at the next word. The layer-norm check confirms the compact backward formula, and the normalized rows have mean 0 and standard deviation 1.

Challenge: three-bit parity

Exercise 10. The parity of three bits is 1 when an odd number of them are 1. It is the generalization of XOR to three inputs and it cannot be solved by a single layer. Train a network with one hidden layer of 8 tanh units and a sigmoid output on all 8 possible inputs, until every one is classified correctly. Write the training loop yourself. Then answer: how many hidden units are enough in principle, and why does the problem get harder as the number of bits grows? See Your First Trained Network.

Python Solution to exercise 10

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three hidden units are enough in principle: each can detect "at least k of the inputs are on" for k = 1, 2, 3, and the output combines them with alternating signs. In general n hidden units suffice for n bits. With more bits the number of input patterns grows as 2^n and the function flips whenever one bit changes, so the network needs either a hidden layer whose width grows with n, or more layers. That is a small, concrete example of why depth matters.

Where to go next

If these ten went smoothly, build something bigger: the projects combine several of these ideas into complete pipelines. If some exercises were hard, return to the lesson linked from them and run its examples again, changing one number at a time. For a quick conceptual review, try the Deep Learning Interview Questions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does subtracting the maximum logit before exponentiating not change the softmax result?

    exp(z - c) / sum(exp(z - c)) equals exp(z) / sum(exp(z)). The shift only keeps the numbers in a safe range.
  2. 2. What does a relative error around 1e-10 in a gradient check tell you?

    Central differences with a small step are accurate to roughly 1e-9 or better. A match at that level means the formula and code agree.
  3. 3. In inverted dropout, why are the surviving activations divided by (1 - p) during training?

    If a fraction p of units is zeroed, the survivors carry the whole signal, so scaling by 1/(1-p) keeps the average output unchanged and lets evaluation skip dropout entirely.
  4. 4. What is the effect of the causal mask in attention?

    Masked scores become minus infinity before softmax, so their weights are exactly zero.

Summary

  • Subtract the maximum logit for a stable softmax; compute cross-entropy from log-probabilities.
  • A dense layer from a to b units has a * b + b parameters; check shapes batch-first.
  • Always verify a hand-written backward pass with finite differences before training with it.
  • Momentum and Adam cope with narrow valleys that stall plain gradient descent.
  • Inverted dropout scales at training time so evaluation is a no-op.
  • Convolution, causal attention and layer norm each have a short NumPy implementation you can check against a reference or a numeric gradient.
  • Parity shows why hidden layers, and depth, are needed for non-linear functions.

Next: Deep Learning Interview Questions.