Skip to content
Tutorials Center
My learning

Topics

Lesson 72 of 73 ยท Practice and Review

Deep Learning Interview Questions

Fifteen common deep learning interview questions with short answers and runnable NumPy demos: vanishing gradients, batch norm, overfitting, attention cost and more.

Deep learning interviews tend to circle around the same ideas: why does a network need non-linearities, why do gradients vanish, what does batch normalization actually do, how do you fight overfitting, and what makes attention expensive. A good answer is short, correct and backed by an example. This page gives you fifteen questions with answers you could say out loud in about a minute, and for the questions that are easy to demonstrate it adds a small runnable program, because being able to show an idea is the strongest way to prove you understand it.

Read the answers as a review, not as a script to memorize. Each one points to the lesson where the topic is explained in depth, so you can follow the link when an answer feels thin.

How to answer

One-line answer Start with the direct answer.
Why Give the mechanism in two sentences.
Example A number, a formula or a tiny piece of code.
Trade-off Say when it stops being true.
1 / 2

Fundamentals

1. What is the difference between machine learning and deep learning?

Machine learning is the broad idea of learning a function from data. Deep learning is the subset that uses neural networks with many layers, which learn their own features from raw inputs (pixels, characters, audio) instead of relying on hand-designed ones. Classical methods such as gradient-boosted trees usually win on small tabular data; deep learning wins when there is a lot of unstructured data and when the structure of the input (grids, sequences) can be built into the architecture. See What Is Deep Learning?.

2. Why does a network need non-linear activation functions?

Without them, a stack of layers collapses into a single linear map: W2 @ (W1 @ x) is just (W2 @ W1) @ x. Depth would add nothing, and the network could never learn XOR or anything curved. Non-linearities such as ReLU let each layer bend the space so that later layers can combine the pieces. See Activation Functions.

3. Why not initialize all weights to zero (or to the same constant)?

Because of symmetry. If every hidden unit starts with identical weights, every unit receives an identical gradient and stays identical forever, so a layer of 100 units behaves like one unit. Random initialization breaks the symmetry, and its scale (Xavier or He) keeps signals from shrinking or exploding. See Weight Initialization.

4. Why use cross-entropy instead of mean squared error for classification?

With a sigmoid or softmax output, MSE multiplies the gradient by the output's own slope, which is almost zero exactly when the model is confidently wrong, so learning stalls. Cross-entropy cancels that factor and gives a gradient of prediction - target, large when the mistake is large. Cross-entropy is also the negative log-likelihood of the labels. See Loss Functions.

The next example shows both ideas: a network that starts with identical weights keeps its columns identical, and the gradient of the two losses when a positive example is predicted with a probability of 0.0003.

Python Symmetry, collapsing layers and loss gradients

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the model is badly wrong (z = -8, a probability of 0.0003), the MSE gradient is about -0.0007: the loss is high, but the signal to fix it is nearly gone. The cross-entropy gradient is -1.0, a strong push.

Training

5. What are vanishing and exploding gradients, and how do you fight them?

During backpropagation the gradient is multiplied by one Jacobian per layer. If those factors are typically below 1 (sigmoid derivatives are at most 0.25), the gradient shrinks exponentially with depth and early layers stop learning; if they are above 1, it explodes. Remedies: ReLU-family activations, careful (He or Xavier) initialization, normalization layers, residual connections, gated cells (LSTM/GRU) for sequences, and gradient clipping for explosions. See Weight Initialization and Backpropagation Through Time.

Python Gradient size across 20 layers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

By the time the gradient reaches the first layer of the sigmoid network it is about 1e-13 of its original size, which means that layer would effectively never learn. The ReLU network with He initialization keeps a gradient between 1.6 and 3.4 at every depth. The third row shows that a ReLU network can also vanish if the weights start too small, which is why initialization matters as much as the activation.

6. What does batch normalization do, and how does it behave at test time?

For each feature it subtracts the mini-batch mean and divides by the mini-batch standard deviation, then applies a learned scale and shift. This keeps activations in a well-behaved range, allows larger learning rates and acts as a mild regularizer, because each example is normalized with statistics that depend on the other examples in its batch. At test time there may be a single example, so the layer uses running averages of the mean and variance collected during training instead of batch statistics. Layer norm normalizes across features within one example, so it needs no batch and is used in transformers. See Batch Norm and Layer Norm.

Python Batch norm: training mode versus evaluation mode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In training mode the output has mean 0 and standard deviation 1 for every batch. The running estimates converge to the true statistics (mean near 5, variance near 4). A single example in evaluation mode is normalized correctly with those frozen numbers, while the same example in training mode becomes all zeros, because a batch of one has no spread. This is the classic bug of forgetting to switch a model to evaluation mode.

7. What is overfitting, how do you detect it, and what can you do about it?

Overfitting means the model fits noise and quirks of the training set, so training error keeps falling while validation error stalls or rises. Detect it by comparing training and validation curves. Remedies, roughly in order of effectiveness: get more data or augment it, simplify the model, add weight decay (L2), add dropout, stop early, and ensemble. See Overfitting, Underfitting and Regularization.

Python Overfitting with polynomials, and L2 as a cure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The degree-15 polynomial has enough freedom to pass near all 20 noisy training points (its training error is the lowest of all) but its test error is far worse than that of the plain cubic. Adding a small L2 penalty on the coefficients, which is exactly weight decay, tames the wild swings and brings the test RMSE from 22.2 back to 0.29, close to the cubic, even though the model has the same number of parameters.

8. When would you choose SGD with momentum over Adam?

Adam adapts the step size per parameter and works well with default settings, which makes it the common first choice and the standard for transformers (as AdamW, with decoupled weight decay). SGD with momentum has less memory overhead and, with a tuned learning rate and schedule, often generalizes slightly better on image classification. Rule of thumb: Adam to get a good result quickly, tuned SGD when the last fraction of a percent matters. See Momentum, RMSProp and Adam.

9. What happens if the learning rate is too high or too low?

Too high: the loss oscillates or diverges (and may become nan). Too low: the loss decreases but painfully slowly, and may stall in a poor region. The best value is usually near the largest one that is still stable, often with warmup and decay. A learning-rate range test (increase the rate exponentially while recording the loss) finds it quickly. See Learning Rate Schedules.

10. How does dropout behave at training versus test time?

During training each unit is zeroed with probability p and the survivors are scaled by 1 / (1 - p) ("inverted dropout"), so the expected activation stays the same. At test time dropout is switched off and nothing needs rescaling. It can be seen as training an ensemble of thinned networks that share weights. See Dropout and Data Augmentation and exercise 6 in Practice Exercises.

Architectures

11. Why do convolutional networks beat fully connected ones on images?

Two reasons: parameter sharing (the same small filter is applied at every position, so a feature learned in one corner is found everywhere) and local connectivity (each output depends on a small neighbourhood). Together they cut the parameter count by orders of magnitude and build in a useful assumption, that image statistics do not depend on position. See Convolution, Pooling and Feature Maps.

12. What are skip (residual) connections for?

A residual block computes x + F(x) instead of F(x). The identity path gives gradients a direct route to earlier layers, so very deep networks train, and each block only has to learn a small correction rather than a full transformation. Transformers use the same trick around attention and feed-forward blocks. See Classic CNN Architectures.

13. What is the cost of self-attention, and why do we divide by the square root of the dimension?

Attention compares every position with every other position, so computing the score matrix takes time and memory proportional to the square of the sequence length (and linear in the head dimension). That is why long contexts are expensive and why efficient-attention research exists. The division by sqrt(d) is needed because the dot product of two random d-dimensional vectors has a standard deviation that grows like sqrt(d); without scaling, the softmax would saturate (one weight near 1, the rest near 0) and its gradients would vanish. See The Attention Mechanism.

Python Attention cost, the sqrt(d) factor and parameter sharing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Quadrupling the sequence length multiplies the score matrix by 16. The standard deviation of the dot products is about 2, 8 and 32 for d = 4, 64 and 1024 (the square roots of 4, 64 and 1024). Without scaling, the largest softmax weight among 10 random keys is 0.20 at d = 4 but 1.00 at d = 1024, which means one key takes everything; with the scaling it is 0.15 and 0.38, soft enough to learn from. The last lines show the saving from sharing: a dense layer from a 224 by 224 RGB image to 1,000 units needs about 150 million weights, while a convolutional layer with 64 filters needs 1,792.

14. How do transformers compare with RNNs?

An RNN processes tokens one by one, so training cannot be parallelized across time and long-range information has to survive many steps. A transformer lets any position attend directly to any other and processes the whole sequence in parallel, which scales much better on modern hardware, at the price of quadratic memory in the sequence length and a need for positional information. RNNs and state-space models still appeal when constant-memory streaming inference matters. See The Transformer Block.

15. What is transfer learning and when does it work?

You start from a network pretrained on a large dataset, keep its early layers (generic features such as edges or word senses), and train a new head, or fine-tune everything with a small learning rate, on your small dataset. It works best when the target domain is related to the pretraining domain and when your own data is limited. If the domains differ a lot, or you have plenty of data, training from scratch can match it. See Transfer Learning and Fine-Tuning.

More questions

16. What is gradient clipping, and when do you need it?

Gradient clipping caps the size of the gradient before the update, either by value (each component limited to a range) or, more commonly, by global norm: if ||g|| exceeds a threshold, scale the whole gradient down to that length. It keeps the direction but prevents one exploding gradient from throwing the weights far away. It is standard for recurrent networks and transformers, where occasional huge gradients appear. See Backpropagation Through Time.

17. What is momentum-based gradient descent?

Momentum keeps a running average of past gradients, a "velocity", and moves along it. Consistent directions build up speed, while components that flip sign from step to step cancel out. That damps oscillation across a narrow ravine and speeds progress along it. Nesterov momentum evaluates the gradient at the point where the velocity is about to take you, which reacts slightly sooner. See Optimizers.

18. What are the kernel, receptive field and padding in a CNN?

A kernel (filter) is the small grid of weights slid across the input. The receptive field of a unit is the region of the original input that can influence it; it grows with every layer, with larger kernels and with strides. Padding adds a border (usually zeros) around the input: "valid" uses none and shrinks the output, "same" keeps the output size equal to the input (at stride 1) and lets edge pixels be seen as often as the rest. See Convolution, Pooling and Feature Maps.

19. What are BiRNNs and BiLSTMs?

A bidirectional RNN runs one recurrent layer forward and another backward over the same sequence and concatenates their states at each step, so every position sees both its past and its future. Swapping in LSTM cells gives a BiLSTM. It needs the whole sequence up front, so it suits tagging and encoding but not live generation. See LSTM and GRU.

20. What is the attention mechanism, and what kinds are there?

Attention lets a model compute, for each output, a weighted average of values where the weights come from how well a query matches each key. Kinds you will meet: additive (Bahdanau, a small network scores the pair) versus dot-product and scaled dot-product (cheap, used in transformers); self-attention (queries, keys and values from the same sequence) versus cross-attention (queries from one sequence, keys and values from another); multi-head attention (several in parallel); and causal (masked) attention for generation. Because attention has no sense of order, positional encodings (sinusoidal, learned or rotary) are added to the inputs. See Self-Attention and Multi-Head Attention.

21. What are the main types of autoencoder?

Undercomplete (a small code forces compression), sparse (a wide code with a sparsity penalty), denoising (reconstruct the clean input from a corrupted one), contractive (penalize the encoder's sensitivity to small input changes) and variational (the code is a distribution, so you can sample new data). See Autoencoders.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A network starts with every weight equal to 0.5. What goes wrong?

    Symmetric weights give symmetric gradients. Random initialization is needed to make the units different.
  2. 2. Why does batch normalization use running statistics at test time?

    At inference the output for one example should not depend on the other examples in a batch.
  3. 3. Training loss keeps falling but validation loss starts to rise. What is the most likely diagnosis?

    The model is fitting details of the training set that do not generalize. Regularize, augment or stop earlier.
  4. 4. By what factor does the attention score matrix grow when the sequence length doubles?

    The matrix has n by n entries, so doubling n multiplies the entries by 2 squared, which is 4.
  5. 5. Why is the gradient of cross-entropy with a sigmoid output equal to prediction minus target?

    The derivative of the log-loss with respect to p contains 1/(p(1-p)), which cancels the sigmoid slope p(1-p) and leaves p - y.

Summary

  • Have a one-line answer, the mechanism, a small example and the caveat ready for each core topic.
  • Non-linearity, random initialization and cross-entropy each exist to prevent a concrete failure (collapse, symmetry, vanishing signal).
  • Vanishing gradients come from products of small factors; ReLU, He initialization, normalization and skip connections fight them.
  • Batch norm uses batch statistics in training and running statistics in evaluation; forgetting the mode switch is a classic bug.
  • Overfitting shows as a gap between training and validation curves; more data, weight decay, dropout and early stopping narrow it.
  • Attention costs quadratic memory in the sequence length, and the scaling by sqrt(d) keeps the softmax trainable.

Next: Deep Learning Cheat Sheet.