PyTorch Tutorial
Learn PyTorch step by step: 45 lessons in 12 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is PyTorch? Meet PyTorch: a Python deep learning library built on tensors and automatic differentiation. Learn eager execution, the ecosystem, install options and how this track runs.
- The Deep Learning Landscape A map of deep learning for PyTorch learners: how it evolved, where it is used, why it is hard, how to cope, and how PyTorch compares with TensorFlow, Keras, Caffe and MXNet.
- Your First Tensors Create PyTorch tensors with tensor, zeros, ones, arange, rand and randn, read shape, dtype and device, set seeds, and compare tensors with NumPy arrays.
Tensors in Depth
- Indexing, Slicing and Reshaping Index and slice PyTorch tensors, use boolean masks, and reshape with view, reshape, permute, squeeze, unsqueeze, cat and stack while understanding shared storage.
- Tensor Math and Broadcasting Do tensor math in PyTorch: elementwise ops, matmul, reductions with dim and keepdim, broadcasting rules, in-place operations and einsum, with live NumPy demos.
- Dtypes, Precision and NumPy Interop Learn PyTorch dtypes (float32, float64, float16, bfloat16, ints), type promotion, to(), and how from_numpy and numpy() share memory, with a live precision demo.
Autograd
- Computational Graphs and the Chain Rule Understand how autograd works: forward and backward passes, computational graphs and the chain rule, walked through by hand and rebuilt in a tiny Python Value class.
- Autograd in Practice Use PyTorch autograd with confidence: requires_grad, backward(), gradient accumulation, zero_grad, no_grad, detach and retain_graph, with live stand-ins.
- Build a Tiny Autograd Tensor Write a NumPy-backed Tensor with add, mul, matmul, relu and sum, a topological backward pass and broadcasting-aware gradients, then train a small network.
- Custom Autograd Functions and Hooks Write a torch.autograd.Function with its own backward, verify it with gradcheck, use tensor hooks, higher-order gradients and torch.func, with live checks.
Building Models
- Neurons, Perceptrons and Feedforward Networks Learn the ideas behind every PyTorch model: the artificial neuron, the perceptron and its limits, multi-layer perceptrons, forward propagation and the main network types.
- Linear Regression the PyTorch Way Fit a line the PyTorch way with data, model, loss, optimizer and the five-line training loop, then run the same loop live in NumPy gradient descent.
- Modules, Parameters and Layers Understand nn.Module, forward, nn.Linear, parameters, named_parameters, state_dict, Sequential, ModuleList and train/eval mode, with a live NumPy layer stack.
- Activation Functions and Loss Functions Learn ReLU, GELU, sigmoid, tanh and softmax, plus MSELoss, CrossEntropyLoss and BCEWithLogitsLoss, logits vs probabilities and numerical stability, live.
- Initialization, Normalization and Dropout Learn how Xavier and Kaiming initialization, BatchNorm, LayerNorm and Dropout keep deep networks trainable, and how train and eval modes change them.
Training
- Optimizers: SGD, Momentum, Adam Understand torch.optim: how SGD, momentum, RMSprop, Adam and AdamW update weights, what weight decay does, and why zero_grad and step order matter.
- Learning Rate Schedules and Gradient Clipping Learn StepLR, cosine, warmup, OneCycle and ReduceLROnPlateau in PyTorch, how clip_grad_norm_ stops exploding gradients, and how to choose a learning rate.
- Datasets and DataLoaders Learn how PyTorch's Dataset and DataLoader feed data to a model: batch_size, shuffle, num_workers, collate_fn, samplers, random_split and IterableDataset.
- The Complete Training Loop Write a complete PyTorch training loop: epochs and batches, train/validation split, correct metric averaging, eval under no_grad, early stopping and logging.
- Overfitting and Regularization Understand bias, variance and overfitting with live examples, then fix them with weight decay, dropout, data augmentation and early stopping in PyTorch.
Computer Vision
- Convolutions and CNNs Learn how convolutions work and how to build a small CNN in PyTorch: Conv2d, kernels, stride, padding, pooling, output shapes and receptive fields.
- CNN Architectures, Pooling and ResNet Tour the landmark CNN designs from LeNet to ResNet, understand pooling types and residual connections, and see why skip connections fix the degradation problem.
- Image Classification Pipeline Build an image classification pipeline: torchvision datasets, transforms, normalization, a small CNN, evaluation and a confusion matrix, with live stand-ins.
- Transfer Learning and Fine-Tuning Reuse pretrained networks in PyTorch: load weights, freeze layers, replace the classifier head, set discriminative learning rates and decide when to fine-tune.
Sequences and Transformers
- RNNs, LSTMs and GRUs Understand recurrent networks in PyTorch: hidden state, nn.RNN, nn.LSTM, nn.GRU, batch_first, padding and packing, and why gradients vanish, with live NumPy stand-ins.
- Embeddings and Text Data Prepare text for PyTorch: tokenization, vocabularies, padding and masks, nn.Embedding lookups, mean pooling and a text classifier skeleton with live NumPy stand-ins.
- Attention from Scratch Understand scaled dot-product attention, queries, keys and values, masks, multi-head attention and positional encodings, with PyTorch code and live NumPy stand-ins.
- Building a Transformer Block Assemble a transformer block in PyTorch: attention, residual connections, LayerNorm, feed-forward layers, a causal LM head and parameter counting.
- Fine-Tuning, PEFT and LoRA Learn how to adapt large pretrained models cheaply: full fine-tuning, parameter-efficient methods, a LoRA layer written in PyTorch, adapters and prompt tuning.
Generative Models
- Autoencoders and Variational Autoencoders Build autoencoders in PyTorch: encoder, bottleneck and decoder, reconstruction loss, denoising, sparse and convolutional variants, and the variational autoencoder.
- Generative Adversarial Networks (GANs) Understand GANs and build one in PyTorch: generator and discriminator, the minimax loss, a training loop, common failures, and variants such as cGAN, WGAN, CycleGAN and StyleGAN.
Saving, Speed and Hardware
- Saving and Loading Models Save and load PyTorch models safely: state_dict, torch.save and torch.load, weights_only, resumable checkpoints with optimizer state, ONNX and TorchScript.
- GPUs, Devices and Mixed Precision Run PyTorch on CPU, CUDA or Apple GPUs: .to(device), pinned memory, autocast, GradScaler, bfloat16 and memory use, with live float16 precision demos.
- Making Training Faster Find and fix PyTorch bottlenecks: profiling, DataLoader workers, torch.compile, channels_last, gradient accumulation and benchmarking, with live NumPy demos.
- Distributed Training Overview Understand PyTorch multi-GPU training: DataParallel versus DistributedDataParallel, process groups, gradient synchronization, FSDP and torchrun, with a NumPy simulation.
Debugging and Practice
- Debugging PyTorch Code Find PyTorch bugs fast: shape and dtype errors, device mismatches, NaN losses, vanishing gradients, anomaly detection, overfitting one batch and seeds.
- Testing and Deploying Models Write unit tests for models and losses, check determinism and eval mode, then learn the options for serving, quantizing and exporting a PyTorch model.
Projects
- Project: A Neural Network Without PyTorch Build a classifier from scratch: a NumPy autograd Tensor, a Module and Linear layers, softmax cross-entropy and Adam, then compare with the PyTorch code.
- Project: Handwritten Digit Classifier Build a handwritten digit classifier end to end: the PyTorch CNN pipeline, then a live scikit-learn version with a confusion matrix and an error gallery.
- Project: Sentiment Classifier Build a sentiment classifier: tokenize, build a vocabulary, pad, embed, pool and train. PyTorch code is read-only, with live NumPy and scikit-learn versions.
- Project: A Character-Level Language Model Build a character-level language model: count a bigram table, train the same model with gradients, add context, and sample text with temperature.
- Project: Forecasting a Time Series Forecast a time series: build sliding windows, set honest baselines, split by time, and compare a PyTorch LSTM (read-only) with a live autoregressive model.
Practice and Review
- Practice Exercises with Solutions Eight PyTorch exercises with worked solutions: tensor shapes, broadcasting, gradients by hand, module design, loop bugs and conv shapes, with live NumPy checks.
- PyTorch Interview Questions Prepare for PyTorch interviews: tensors vs arrays, autograd, zero_grad, eval vs no_grad, DataLoader, BatchNorm, mixed precision and common pitfalls, with demos.
- PyTorch Cheat Sheet A one-page PyTorch cheat sheet: tensor creation, shapes, autograd, nn layers, losses, optimizers, a training loop template, device handling and saving models.