Hugging Face Transformers Tutorial
Learn Hugging Face Transformers step by step: 38 lessons in 9 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is Hugging Face Transformers? Meet the Hugging Face ecosystem: the Transformers library, the Hub, Datasets and Tokenizers, and learn what you can build and how this course teaches it.
- Installation and Environment Setup Install transformers, PyTorch and friends in a clean virtual environment, pick the right device, understand the cache folder and log in to the Hub safely.
- The Transformer Architecture in One Lesson Understand how transformers work: embeddings, self-attention, encoder, decoder and encoder-decoder models, with a runnable NumPy attention example.
- Your First Pipeline Use pipeline() to run a pretrained model in two lines, learn what happens inside, choose models and devices, and build a runnable stand-in of the same flow.
- Model Families and Tasks Map tasks to model families: BERT-style encoders, GPT-style decoders, T5 and BART, vision transformers and Whisper, and learn how to choose the right size.
Hub, Models and Tokenizers
- The Model Hub: Repositories, Model Cards and Versions Learn how the Hugging Face Hub works: Git-based repositories, model cards, files, revisions, searching and downloading models safely with from_pretrained.
- Tokenization Basics Learn why models need tokenizers, how text becomes token ids, and the trade-offs between word, character and subword tokenization, with runnable examples.
- Subword Tokenizers: BPE and WordPiece Learn how Byte-Pair Encoding learns merges from a corpus and how WordPiece splits words greedily, with runnable Python implementations you can edit.
- AutoTokenizer in Practice Use AutoTokenizer for batches: padding, truncation, attention masks, token type ids, offsets, pairs of texts and chat templates, with runnable stand-ins.
- AutoModel, Configs and Task Heads Understand AutoConfig, AutoModel and the AutoModelFor classes: body versus head, logits, softmax, label mappings, and how from_pretrained builds a model.
Pipelines and Inference
- Pipelines for Text Tasks Use text pipelines for classification, zero-shot labels, NER, question answering, fill-mask, summarization and translation, and learn to read their outputs.
- Pipelines for Vision and Audio Run image classification, object detection, speech recognition and audio classification pipelines, and see how images become patches and audio becomes frames.
- Text Generation and Decoding Strategies Understand greedy search, beam search, temperature, top-k and top-p sampling, repetition penalties and stopping, with runnable NumPy demos and generate() settings.
- Batching, Devices and Inference Speed Speed up inference with batching, sorting by length, half precision, GPU placement, KV caching and memory arithmetic, with runnable estimators.
Datasets and Preprocessing
- Loading Datasets with the datasets Library Load datasets from the Hub and local files with load_dataset, understand Dataset and DatasetDict, Arrow memory mapping, features, indexing, filter and select.
- Preprocessing with Dataset.map Tokenize datasets with Dataset.map, use batched processing, remove columns, handle long texts with overflowing chunks and align token labels.
- Splits, Streaming and Custom Data Create reliable train, validation and test splits, avoid data leakage, stream huge datasets, build a Dataset from your own records and push it to the Hub.
Fine-tuning with Trainer
- The Trainer and TrainingArguments Learn how the Hugging Face Trainer runs the training loop for you: TrainingArguments, steps and epochs, evaluation, logging, learning-rate schedules, with runnable stand-ins.
- Data Collators and Dynamic Padding Understand how data collators batch tokenized examples: dynamic padding, attention masks, label padding, masked language modeling and length grouping, with runnable stand-ins.
- Custom Training Loops with Accelerate Write your own fine-tuning loop when Trainer is not enough: gradient accumulation, clipping, schedulers and Hugging Face Accelerate for multi-GPU, with runnable stand-ins.
- Fine-tuning for Text Classification Fine-tune a pretrained transformer for text classification: classification head, label mappings, logits and softmax, freezing layers and a complete Trainer script, with stand-ins.
- Checkpoints, Callbacks and Early Stopping Save, resume and manage training checkpoints, use Trainer callbacks and EarlyStoppingCallback, keep the best model and avoid overfitting, with runnable stand-ins.
Efficient Fine-tuning and Quantization
- Parameter-Efficient Fine-Tuning (PEFT) Overview Why full fine-tuning gets expensive and how parameter-efficient methods (adapters, LoRA, prefix tuning, prompt tuning, BitFit) cut trainable parameters, with runnable counting stand-ins.
- LoRA Explained: Low-Rank Adaptation Understand LoRA step by step: the low-rank update W + BA, rank and alpha, target modules, zero initialization and merging, with numpy demos and the peft code you would use.
- Quantization Fundamentals Learn how quantization shrinks models: bits and number formats, absmax and zero-point schemes, per-tensor versus per-channel scales, outliers and error, with runnable numpy demos.
- QLoRA and bitsandbytes: 4-bit Fine-tuning Fine-tune large language models on one GPU with QLoRA: 4-bit NF4 quantization, double quantization, paged optimizers and the BitsAndBytesConfig code, with numpy demos of the ideas.
- Memory Math and Mixed Precision Training Estimate GPU memory for training and inference, understand fp16 versus bf16, loss scaling, gradient checkpointing and batch size trade-offs, with runnable calculators.
Evaluation and Metrics
- The Evaluate Library and Metric Basics Compute metrics with Hugging Face Evaluate: load, compute, add_batch, combine, compute_metrics for Trainer, and how a metric works inside, with a runnable mini version.
- Classification Metrics: Precision, Recall, F1 Evaluate classifiers properly: confusion matrix, accuracy pitfalls, precision, recall, F1, macro versus micro averaging, thresholds and a Trainer compute_metrics function.
- Text Generation Metrics: BLEU, ROUGE, Perplexity Learn how BLEU, ROUGE, perplexity, exact match and token F1 are computed, what they miss, and how to evaluate generated text, with runnable pure-Python implementations.
- Error Analysis, Calibration and Benchmarks Go beyond one accuracy number: slice errors, read confusion matrices, measure calibration with ECE, and learn how benchmark scores for language models are produced and misread.
Deployment and Spaces
- Saving, Pushing and Documenting Models Save a fine-tuned model with save_pretrained, understand the safetensors file format, push to the Hugging Face Hub, and write a model card that makes your model usable and safe.
- Serving a Model with FastAPI and Batching Wrap a Hugging Face pipeline in a FastAPI service, load it once, validate input, and use dynamic micro-batching to raise throughput. Includes runnable latency simulations.
- Gradio Demos and Hugging Face Spaces Turn a Transformers pipeline into a shareable web demo with Gradio and host it free on Hugging Face Spaces: app files, requirements, secrets, hardware, caching and limits.
- Optimized Inference: ONNX, Optimum and Compiled Models Make Transformers inference cheaper: export to ONNX, use Optimum and ONNX Runtime, fuse operators, quantize weights, compile with torch.compile, and benchmark honestly.
Projects and Practice
- Project: A Support-Ticket Classifier, Data to Demo Build a complete text-classification project: data splits, tokenization, a baseline, fine-tuning with Trainer, error analysis, saving and a demo, with a runnable numpy baseline.
- Project: Semantic Search with Embeddings Build a semantic search engine: chunk documents, embed with mean pooling, rank by cosine similarity, evaluate recall@k, and scale up. Real code plus a runnable numpy stand-in.
- Troubleshooting and Practice A field guide to common Transformers problems (shape errors, NaN loss, out of memory, wrong labels, slow training), a debugging routine, a loss-curve doctor, and practice tasks.