Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 45 ยท Practice and Review

PyTorch Interview Questions

Prepare for PyTorch interviews: tensors vs arrays, autograd, zero_grad, eval vs no_grad, DataLoader, BatchNorm, mixed precision and common pitfalls, with demos.

Interviewers rarely ask you to recite an API. They ask why something works the way it does, and whether you have been bitten by the classic mistakes. This page collects questions that come up again and again for machine learning engineer and research engineer roles, with answers short enough to say out loud and detailed enough to show you understand the machinery.

The PyTorch snippets here are read-only, because the library cannot run on this site. Three questions have live NumPy demos that reproduce the behavior being asked about (gradient accumulation, train versus eval mode, and half-precision limits). They are stand-ins that show the mechanism, not PyTorch itself. If an answer feels unfamiliar, the linked lesson explains it in full.

What interviewers probe

Tensors Shapes, views, dtypes, devices.
Autograd Graphs, accumulation, no_grad.
Training Loops, optimizers, schedules.
Pitfalls Bugs you have actually debugged.

Say why, not only what.

1 / 2

Tensors and data

What is the difference between a PyTorch tensor and a NumPy array?

Both are n-dimensional arrays with similar indexing and broadcasting. A tensor adds three things: it can live on a GPU (or other accelerator), it can record operations for automatic differentiation (requires_grad), and it plugs into torch.nn and the optimizers. torch.from_numpy(a) and t.numpy() share memory on the CPU, so changing one changes the other. See Dtypes, Precision and NumPy Interop. A tensor that requires grad must be detached (t.detach().cpu().numpy()) before you convert it.

What is the difference between view and reshape?

view never copies: it returns a tensor sharing the same storage, and it only works when the memory layout allows the new shape (the tensor must be compatible with the requested strides, which is why it often fails after transpose or permute). reshape returns a view when it can and silently copies when it must. Use view when you want a guarantee of no copy, and reshape when you just want the shape. .contiguous() forces a standard memory layout. Covered in Indexing, Slicing and Reshaping.

How does broadcasting work?

Shapes are aligned from the right. Each pair of dimensions must be equal or one of them must be 1; a missing dimension counts as 1. Size-1 dimensions are stretched virtually, without copying data. A common bug is subtracting a (N,) vector from an (N, 1) tensor and silently getting an (N, N) result. The rule is the same as in NumPy: Tensor Math and Broadcasting.

Why does x += 1 sometimes break training but x = x + 1 does not?

In-place operations overwrite the tensor's data. If autograd saved that tensor to compute a gradient later, the saved value is gone, and backward() raises an error about a variable modified by an in-place operation. The out-of-place form creates a new tensor and leaves the saved one intact. In-place operations on a leaf that requires grad are rejected outright.

Autograd

How does autograd work?

During the forward pass, PyTorch records every operation on tensors that require gradients as a node in a directed acyclic graph, storing whatever it needs for the backward formula. loss.backward() walks that graph in reverse topological order, applying the chain rule at each node and adding results into the .grad of every leaf tensor. The graph is rebuilt on each forward pass (a dynamic or define-by-run graph) and freed after backward() unless you pass retain_graph=True. See Computational Graphs and the Chain Rule.

Why do we call optimizer.zero_grad()?

Because backward() accumulates: it adds new gradients into .grad instead of overwriting. That is deliberate. It makes gradient accumulation across micro-batches trivial and supports tensors used in several places. But in an ordinary loop, forgetting zero_grad() makes each step use the sum of all earlier gradients. Modern code calls optimizer.zero_grad() at the start of each iteration (it sets gradients to None by default, which is slightly cheaper than filling zeros). The first demo shows the accumulation.

Python Why zero_grad exists (NumPy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What is the difference between torch.no_grad(), model.eval() and inference_mode?

They solve different problems and are usually used together at evaluation time.

  • model.eval() changes the behavior of layers: dropout is switched off and batch norm uses running statistics. It does not stop gradient tracking.
  • torch.no_grad() stops recording the graph, saving memory and time. It does not change layer behavior.
  • torch.inference_mode() is a stricter, faster version of no_grad for code that will never need gradients; tensors made inside cannot be used in autograd later.

A validation loop therefore needs model.eval() and a with torch.no_grad(): block, and then model.train() again afterwards. Forgetting eval() gives noisy, wrong validation numbers; forgetting no_grad() just wastes memory. More in Autograd in Practice.

What does detach() do, and what is a leaf tensor?

detach() returns a tensor that shares data with the original but is cut out of the graph, so no gradient flows through it. A leaf tensor is one created directly by the user (a parameter or an input), rather than as the result of an operation. Only leaves that require grad get .grad filled in by default; for a non-leaf you must call retain_grad().

Models and layers

Why does CrossEntropyLoss take logits and not probabilities?

nn.CrossEntropyLoss applies log_softmax internally and computes the negative log-likelihood in a numerically stable way (subtracting the max before exponentiating). Feeding it softmax outputs applies softmax twice, which squashes gradients and trains badly. For binary tasks the same logic applies to BCEWithLogitsLoss versus BCELoss after a sigmoid. See Activation Functions and Loss Functions.

How do BatchNorm and Dropout behave differently in training and evaluation?

In training, batch norm normalizes with the statistics of the current batch and updates running averages; in evaluation it uses the stored running averages, so a single example gives a stable output. Dropout randomly zeros activations in training and rescales the survivors by 1 / (1 - p) (so the expected value is unchanged); in evaluation it does nothing. This is why model.train() and model.eval() matter. The second demo shows both.

Python Train vs eval mode (NumPy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What is the difference between nn.Module, parameters() and state_dict()?

nn.Module is the base class for layers and models. Any nn.Parameter or sub-module assigned as an attribute is registered automatically, so parameters() (what you hand to the optimizer) finds it. state_dict() is an ordered dictionary of tensors, parameters plus buffers such as batch-norm running statistics, which is what you save and load. A plain tensor stored as an attribute is not registered and will not be saved or moved by .to(device); use register_buffer for such state. See Modules, Parameters and Layers.

Why do we initialize weights carefully?

If every layer shrinks or inflates the signal a little, the effect compounds with depth: activations and gradients vanish or explode. Xavier initialization suits tanh and sigmoid, Kaiming initialization suits ReLU, and both pick a weight variance from the layer's fan-in and fan-out so the signal keeps roughly the same scale. Normalization layers reduce this sensitivity but do not remove the need for a sensible start. See Initialization, Normalization and Dropout.

Data and training

What does a DataLoader do, and how do you make it faster?

A DataLoader wraps a Dataset (which defines __len__ and __getitem__) and turns it into an iterator of batches: it samples indices, loads items, and combines them with a collate_fn. shuffle=True reshuffles every epoch. To speed it up, use num_workers > 0 so loading and augmentation happen in parallel processes, pin_memory=True with non_blocking=True copies for GPU training, and persistent_workers=True to avoid restarting workers each epoch. If the GPU sits idle, the loader is usually the bottleneck. See Datasets and DataLoaders.

Write the standard training step. What is the order of calls?

Python
model.train()
for x, y in loader:
    x, y = x.to(device), y.to(device)
    optimizer.zero_grad()
    loss = loss_fn(model(x), y)
    loss.backward()
    optimizer.step()
    scheduler.step()      # per batch for OneCycle, per epoch for StepLR

zero_grad must come before backward (anywhere earlier in the iteration is fine), and step must come after it. Swapping scheduler.step() before optimizer.step() triggers a warning because the first learning rate is skipped. See The Complete Training Loop.

How do you debug a loss that does not go down?

Work from cheap to expensive. First, overfit a single small batch: if the model cannot reach near-zero loss on 8 examples, there is a bug (wrong loss input, labels misaligned, missing zero_grad, frozen parameters). Then check the data: print shapes, label ranges and a few decoded samples. Check the learning rate (try 10x up and down), confirm gradients are non-zero and finite, and make sure model.train() is set. For nan losses, look for division by zero, log(0), or an exploding learning rate, and consider gradient clipping.

How do you guard against exploding gradients?

Clip the global gradient norm with torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) after backward() and before step(). Also lower the learning rate, add warmup, use normalization layers and a sensible initialization. See Learning Rate Schedules and Gradient Clipping.

Performance and hardware

What is mixed precision, and why use GradScaler?

Mixed precision runs most operations in a 16-bit type (float16 or bfloat16) for speed and lower memory, while keeping sensitive operations and the master weights in float32. torch.autocast chooses the dtype per operation. float16 has a narrow range, so small gradients can underflow to zero, which is what GradScaler fixes by multiplying the loss by a scale factor and unscaling before the update. bfloat16 has the same range as float32 (with less precision), so it normally needs no scaler. The third demo shows why float16 is delicate.

Python Why float16 is delicate (NumPy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What does torch.compile do?

It captures your model's Python code into graphs and compiles them into fused, optimized kernels, which can remove overhead and memory traffic. The first call is slow because of compilation, and code with heavy data-dependent Python control flow may cause graph breaks that reduce the benefit. It is a one-line change, model = torch.compile(model), but measure before and after. See Making Training Faster.

What is the difference between DataParallel and DistributedDataParallel?

DataParallel runs in one process and splits each batch across GPUs with Python threads, replicating the model every step, so it is simple but slow. DistributedDataParallel (DDP) runs one process per GPU, each with its own model replica and data shard, and averages gradients with an all-reduce during backward(). DDP is the recommended choice, even on a single machine, and is launched with torchrun. See Distributed Training Overview.

Deep learning concepts

What is the difference between a perceptron and a multi-layer perceptron?

A perceptron is a single neuron with a step activation and can only separate classes with a straight line, so it cannot learn XOR. A multi-layer perceptron adds hidden layers with non-linear activations, which lets it model non-linear relationships and, with enough units, approximate any continuous function. It is trained with backpropagation. See Neurons, Perceptrons and Feedforward Networks.

What are forward and backward propagation?

Forward propagation pushes an input through the layers to produce a prediction and the loss. Backward propagation then applies the chain rule from the loss back through the same graph, computing the gradient of the loss with respect to every parameter. In PyTorch model(x) does the first and loss.backward() the second. See Computational Graphs and the Chain Rule.

What is the bias-variance tradeoff, and how do overfitting and underfitting relate to it?

Bias is error from a model that is too simple to capture the pattern (underfitting: high training and validation error). Variance is error from a model so flexible that it fits noise (overfitting: low training error, high validation error). Reducing one usually raises the other. Remedies for overfitting: more data, augmentation, dropout, weight decay, early stopping, a smaller model. For underfitting: a bigger model, better features, longer training. See Overfitting and Regularization.

What is momentum, and how do adaptive optimizers differ?

Momentum keeps a running average of past gradients, so steps accelerate along consistent directions and damp oscillations. Adaptive optimizers (Adagrad, RMSprop, Adam) also scale each parameter's step by a running measure of its gradient size. Adam combines both. See Optimizers.

What is gradient clipping and when do you need it?

Clipping rescales gradients whose norm exceeds a threshold (clip_grad_norm_) or clamps each entry (clip_grad_value_). It prevents a single huge gradient from destroying the weights, and is standard for RNNs and transformers. See Learning Rate Schedules and Gradient Clipping.

What is batch normalization, and what problem was it designed for?

It normalizes each feature over the mini-batch and then applies a learned scale and shift. It was designed to reduce internal covariate shift, the drifting input distribution each layer sees as earlier layers change, and in practice allows higher learning rates, reduces sensitivity to initialization and regularizes slightly. At evaluation it uses running statistics. See Initialization, Normalization and Dropout.

In a CNN, what are a kernel, a receptive field and padding?

A kernel (filter) is the small weight grid slid over the input. The receptive field of a unit is the region of the original input that can influence it; it grows with depth, stride and pooling. Padding adds a border (usually zeros) so edges are covered and the output size can be preserved ("same" padding). Other options include reflection and replication padding. See Convolutions and CNNs.

What problem does ResNet solve?

Plain networks beyond a certain depth got worse even on the training set (the degradation problem). ResNet adds skip connections so each block computes F(x) + x; learning the identity becomes easy and gradients flow directly through the shortcut. See CNN Architectures, Pooling and ResNet.

Why do plain RNNs have vanishing and exploding gradients, and what are BiRNNs and BiLSTMs?

Backpropagation through time multiplies many Jacobians, so gradients shrink or grow exponentially with the sequence length. LSTM and GRU gates create additive paths that fix vanishing, and clipping handles exploding. A bidirectional RNN or LSTM runs one network forward and one backward and concatenates the states so every position sees both its past and its future. See Recurrent Networks.

What are tokens and embeddings?

A token is a unit of text (a word, subword or character) mapped to an integer ID. An embedding maps each ID to a learned dense vector, where similar tokens end up close together (nn.Embedding). See Embeddings and Text.

What is attention, what is positional encoding, and how do the encoder-decoder and BERT versus GPT designs differ?

Attention lets each position take a weighted average of all positions, with weights from query-key similarity. It has no sense of order, so positional encodings (sinusoidal or learned) are added to the embeddings. An encoder-decoder model encodes the input and decodes the output while cross-attending to the encoder. BERT is encoder-only and bidirectional (good at understanding); GPT is decoder-only and causal (good at generation). See Attention and Transformers.

What is an autoencoder, which types exist, and how does it differ from PCA?

An autoencoder compresses its input to a latent code and reconstructs it. Types: undercomplete, denoising, sparse, convolutional, contractive and variational (VAE). A linear autoencoder spans the same subspace as PCA; non-linear ones capture curved structure. A VAE outputs a distribution (mu, logvar), uses the reparameterization trick and a KL term, and can generate new data. See Autoencoders.

What is a seq2seq model?

An encoder compresses an input sequence into a representation and a decoder generates an output sequence of possibly different length from it: translation, summarization, speech recognition. Early versions used RNNs; modern ones use transformers.

What is a GAN, and what are its main variants?

A generator maps noise to samples and a discriminator tells real from fake; they train in a minimax game. Variants: conditional GAN (label-controlled), DCGAN (convolutional), WGAN (Wasserstein loss, steadier), CycleGAN (unpaired image translation), Progressive GAN and StyleGAN (high-resolution faces with style-based control). Typical problems are mode collapse and unstable training. See GANs.

What is LoRA?

Low-rank adaptation freezes a pretrained weight W0 and trains a low-rank update B @ A (rank r much smaller than the matrix size), with B initialized to zero. It trains well under 1 percent of the parameters, produces small per-task files and can be merged into W0 for inference. See Fine-Tuning, PEFT and LoRA.

Common pitfalls

What mistakes do people make most often?

  • Calling loss.backward() without zero_grad(), or storing loss (not loss.item()) in a list, which keeps every graph alive and leaks memory.
  • Forgetting model.eval() and torch.no_grad() during validation.
  • Passing softmax outputs to CrossEntropyLoss.
  • Moving the model to the GPU but leaving data on the CPU (a device mismatch error), or creating the optimizer before moving the model.
  • Using the wrong dtype: class labels must be long for CrossEntropyLoss, and mixing float64 with float32 raises errors.
  • Shape bugs from broadcasting, such as a (N,) target against an (N, 1) prediction.
  • Not setting seeds, then being unable to reproduce a run. Set torch.manual_seed, and for strict determinism also seed NumPy and Python and enable deterministic algorithms; see Debugging PyTorch Code.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A validation loop gives noisy results although it runs under torch.no_grad(). What is the most likely missing line?

    no_grad only stops graph recording. Dropout and batch norm keep their training behavior until you call model.eval().
  2. 2. Why does PyTorch accumulate gradients in .grad instead of overwriting them?

    Adding into .grad is a design choice: contributions from several backward passes or several uses of a tensor sum correctly. The price is that you must reset gradients each iteration.
  3. 3. What should you feed to nn.CrossEntropyLoss?

    The loss applies log_softmax internally and expects raw logits with class indices (long) as targets.
  4. 4. Which statement about float16 and bfloat16 is correct?

    bfloat16 keeps the 8-bit exponent of float32, so it rarely overflows or underflows, but has fewer mantissa bits. float16 has a narrow range, which is why a GradScaler is used.

Summary

  • Tensors add GPU support and autograd to the array idea; view shares memory, reshape copies only when it must.
  • Autograd records a dynamic graph, backward() accumulates into .grad, so reset gradients every iteration.
  • eval() changes layer behavior, no_grad() stops graph recording, and validation needs both.
  • Pass raw logits to CrossEntropyLoss, and mind BatchNorm and Dropout modes.
  • Debug by overfitting one batch, checking data, learning rate and gradients.
  • Mixed precision needs a scaler for float16, rarely for bfloat16; prefer DDP over DataParallel.

Finish with the PyTorch Cheat Sheet.