Skip to content
Tutorials Center
My learning

Topics

Lesson 73 of 73 ยท Practice and Review

Deep Learning Cheat Sheet

A one-page deep learning reference: core formulas, tensor shapes, activation and loss pairings, optimizer defaults, a debugging checklist and an architecture chooser.

This page collects what you reach for most often while building networks: the formulas, the shapes, the pairings that are almost always right, sensible default settings, and the order in which to debug a run that will not learn. It is a reference, not a lesson, so each part is short and points to the full explanation.

Three runnable checks are included, because a cheat sheet you can test is a cheat sheet you can trust: a shape tracker, a check that the recommended loss pairings are numerically safe, and a run of the optimizer defaults.

The whole field on one screen

Forward Layers turn inputs into predictions.
Loss One number says how wrong they are.
Backward The chain rule gives every gradient.
Update The optimizer nudges the weights.

Every model in this tutorial is this loop with different parts plugged in.

1 / 2

Core formulas

Idea Formula
Dense layer z = X @ W + b, then a = f(z)
Gradient descent step w <- w - lr * dL/dw
Sigmoid 1 / (1 + exp(-z)), derivative s * (1 - s)
Tanh tanh(z), derivative 1 - tanh(z)^2
ReLU max(0, z), derivative 1 where z > 0, else 0
Softmax exp(z_i) / sum_j exp(z_j) (subtract max(z) first)
MSE mean((y_hat - y)^2)
Cross-entropy -mean(log p[correct class])
Softmax + cross-entropy gradient (p - one_hot(y)) / N with respect to the logits
L2 penalty adds lambda * w to each weight gradient
Scaled attention softmax(Q @ K.T / sqrt(d_k)) @ V
Normalization (x - mean) / sqrt(var + eps) * gamma + beta

The pairing in the middle of the table is the most useful identity in the field: when you combine softmax with cross-entropy, the backward pass collapses to "probabilities minus targets". See Backpropagation in NumPy for the derivation and Loss Functions for the stable log-softmax.

Shapes

Batches are rows. If N is the batch size, a dense layer maps (N, in) to (N, out) using W of shape (in, out) and b of shape (out,). Parameter count is in * out + out. Gradients always have the same shape as the thing they belong to.

For images the layout is (N, C, H, W). A convolution with kernel size k, padding p and stride s turns a side length n into floor((n + 2p - k) / s) + 1, and has C_out * (C_in * k * k + 1) parameters. Sequences are (N, T, D) with T time steps. Attention works on (N, heads, T, d_head), and its score matrix is (T, T) per head, which is why its cost grows with the square of the sequence length. See The Convolution Operation and Convolution, Pooling and Feature Maps.

The first example turns these rules into a function you can reuse: it walks through a small network and prints every shape and parameter count.

Python A shape and parameter tracker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Activation and loss pairings

The choice of output layer follows from the task, and the loss follows from the output layer. Hidden layers are a separate decision.

Task Output activation Loss Notes
Regression none (linear) MSE, or Huber if outliers Scale targets to roughly mean 0, std 1
Binary classification sigmoid binary cross-entropy One output unit
Multi-class (one label) softmax categorical cross-entropy One unit per class
Multi-label (several labels) sigmoid on each binary cross-entropy per label Labels are independent
Next-token prediction softmax over vocabulary cross-entropy Perplexity is exp(loss)

For hidden layers, ReLU (or leaky ReLU) is the default for plain networks and CNNs, and GELU is the default inside transformers. Tanh and sigmoid inside hidden layers are mostly historical, except in the gates of recurrent cells. See Activation Functions.

In practice you rarely compute the output activation and the loss separately. Frameworks fuse them into one "loss from logits" function because the fused version is stable for large inputs. The second example shows why.

Python Why the fused loss is the safe one

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Optimizer defaults

Start with these values and change them only when the loss curve tells you to. See Momentum, RMSProp and Adam and Learning Rate Schedules.

Optimizer Typical defaults Use it when
SGD lr 0.01 to 0.1 Small, well-conditioned problems
SGD + momentum lr 0.01 to 0.1, momentum 0.9 CNNs trained with a schedule
RMSProp lr 0.001, rho 0.9 Recurrent networks, older recipes
Adam lr 0.001, betas 0.9 and 0.999, eps 1e-8 First choice for most new problems
AdamW lr 0.0003 to 0.001, weight_decay 0.01 Transformers and anything with weight decay

Other defaults worth remembering: batch size 32 to 256 for most tasks, He initialization for ReLU layers and Xavier for tanh or sigmoid layers, gradient clipping at a norm of 1.0 for recurrent models and transformers, dropout of 0.1 to 0.5, warmup for the first few percent of steps when training transformers, and early stopping on validation loss.

The third example runs the first four rows on the same ill-conditioned bowl, a function that is steep in one direction and shallow in the other, using the textbook defaults.

Python Optimizer defaults on one surface

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The learning rates above are tuned for this particular bowl so that each method behaves sensibly. The table gives starting points, not guarantees: a learning rate range test is still the quickest way to find the right order of magnitude.

Debugging checklist

Work down this list in order and stop at the first item that fails. The full routine, with code, is in Debugging Training Runs.

  1. Data: look at a few raw examples and labels. Check that inputs are scaled, that the train and validation sets do not overlap, and that labels match inputs.
  2. Shapes: print the shape at every layer once. Watch for accidental broadcasting, such as a (N,) array combined with a (N, 1) array.
  3. Starting loss: it should be near ln(K) for K classes, or near the variance of the targets for regression.
  4. Overfit one batch: 16 examples, a few hundred steps. The loss must go to nearly zero.
  5. Gradients: compare against finite differences on a tiny network (Gradient Checking).
  6. Learning rate: try a factor of 10 up and down. This is the most common cause of both flat and exploding losses.
  7. Train and eval modes: dropout and normalization behave differently in each. A model left in the wrong mode gives odd validation numbers.
  8. Curves: train falling while validation rises means overfitting, so add data, augmentation, dropout or weight decay. Both flat and high means underfitting, so use a bigger model, train longer or raise the learning rate.
  9. Metric: accuracy hides problems on imbalanced data. Check precision, recall and the confusion matrix (Evaluating Models).

Architecture chooser

Your data Start with Why
Tabular rows, a few hundred features Gradient boosting first, then an MLP Trees are strong baselines on tables
Small images, simple patterns Small CNN Weight sharing fits local patterns
Photos, limited labels Pretrained CNN, fine-tuned Transfer learning needs far less data
Short time series MLP on windows, or an LSTM or GRU Compare both against a naive baseline
Text, many examples Transformer encoder Attention captures long-range context
Text, generate new text Decoder-only transformer Causal masking matches left-to-right generation
Unlabeled data, compression, anomalies Autoencoder Reconstruction error flags the unusual
Images or audio to generate VAE, GAN or diffusion model Learn the data distribution itself

A good habit before picking anything fancy: build the simplest baseline (a mean predictor, logistic regression, a naive forecast), measure it, and only keep a deeper model if it clearly beats it. See What Is Deep Learning? for when deep learning is the right tool at all.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You train a 10-class classifier and the very first loss prints 0.002. What is the most likely conclusion?

    With sensible initialization the outputs are close to uniform, so cross-entropy starts near ln(K). A tiny first loss usually means a bug such as leaked labels or the wrong loss.
  2. 2. Which output layer and loss fit a task where each image can have several tags at once (for example, 'beach' and 'sunset')?

    Multi-label tags are independent, so each output gets its own sigmoid and its own binary cross-entropy. Softmax would force the probabilities to compete.
  3. 3. A convolution takes a 32x32 input with kernel 5, padding 2 and stride 1. What is the output side length?

    floor((32 + 2*2 - 5) / 1) + 1 = 32. Padding of (k - 1) / 2 keeps the size unchanged at stride 1.
  4. 4. Why do frameworks offer a fused 'cross-entropy from logits' function?

    Computing softmax and then log separately can overflow or produce log(0). The fused form uses the log-sum-exp trick and stays stable.

Summary

  • Every network is the same loop: forward, loss, backward, update. Different tasks only change the parts.
  • Match the output activation to the task, then use the cross-entropy or MSE loss that goes with it, in its fused, stable form.
  • Batches are rows; gradients have the shape of their parameters; convolution output side is floor((n + 2p - k) / s) + 1.
  • Adam with a learning rate of 0.001 is the usual first choice, AdamW for transformers, and SGD with momentum for CNNs with a schedule.
  • Debug in order: data, shapes, starting loss, overfit one batch, gradients, learning rate, modes, curves, metrics.
  • Pick the simplest architecture that suits the data, and beat a simple baseline before keeping something larger.

Next, test yourself with Deep Learning Interview Questions.