Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 38 ยท Projects and Practice

Troubleshooting and Practice

A field guide to common Transformers problems (shape errors, NaN loss, out of memory, wrong labels, slow training), a debugging routine, a loss-curve doctor, and practice tasks.

Most hours spent with Transformers go into figuring out why something did not work. The good news is that failures repeat: the same dozen causes explain most of them. This lesson gives you a debugging routine, a symptom table, a runnable "loss-curve doctor" and a set of practice tasks that cover the whole course.

Real training runs are not possible on this site, so the examples are numpy stand-ins that reproduce the symptoms (a model that overfits one batch, a loss curve that diverges) rather than real Transformers output.

A debugging routine that works

Follow this order. Each step removes a whole class of problems before you spend GPU hours.

  1. Print one batch. Decode the input_ids back to text with tokenizer.decode, check the labels, shapes and padding. Many bugs are visible here: labels shifted, missing special tokens, texts truncated to 3 tokens.
  2. Run a forward pass on a tiny batch on CPU. Shape errors and wrong label counts show up immediately with readable tracebacks (GPU errors are asynchronous and cryptic; CUDA_LAUNCH_BLOCKING=1 helps).
  3. Overfit 8 to 32 examples. A correct pipeline can drive the training loss to near zero on a tiny set. If it cannot, there is a bug in data, labels, loss or learning rate, not a "need more data" problem.
  4. Check the loss at step 0. For K balanced classes, cross-entropy should start near ln(K) (0.69 for 2 classes, 1.10 for 3). Much higher means a wrong head or labels; exactly 0 means labels leaked.
  5. Train on the full set and watch train and validation loss together.
  6. Compare with a baseline (majority class, keyword rules) so you know whether the result is meaningful.
Python Sanity check: overfit a tiny batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The symptom table

Symptom Usual cause What to do
CUDA out of memory Batch, sequence or model too big Smaller per-device batch plus gradient accumulation, bf16, gradient checkpointing, shorter max_length (memory math)
Expected input batch_size to match target batch_size Labels and inputs have different lengths Check that the collator keeps labels; remove columns the model does not accept
Target N is out of bounds A label id is not below num_labels Build labels with ClassLabel, pass num_labels / id2label
ValueError: expected sequence of equal length Padding missing Use DataCollatorWithPadding or padding=True (collators)
Loss is nan Learning rate too high, fp16 overflow, bad input (empty or inf) Lower the LR, switch to bf16, warm up, clip gradients (max_grad_norm)
Loss does not move LR too low, frozen layers, shuffled labels Check requires_grad, labels versus text, optimizer parameters
Train loss falls, validation rises Overfitting Early stopping, more data, augmentation, higher weight decay, smaller model (checkpoints and early stopping)
Accuracy equals the majority-class share Predicts one class Check class balance, LR, and that labels reach the loss
Great validation score, poor real-world results Leakage or distribution shift Deduplicate, split by group or time, build a golden set from real traffic
Different results every run Unseeded randomness set_seed, record versions; remember GPU kernels can be non-deterministic
Warning: "newly initialized weights" A new head was added Expected for fine-tuning; it is trained on your data
Garbled text in generation Wrong tokenizer or missing chat template Load tokenizer from the same repo; use apply_chat_template
Slow tokenization or training Python-level loops, no batching Use map(batched=True), fast tokenizers, num_proc, sorted batches (batching and speed)

Reading a loss curve

Curves speak. Compare training and validation loss across steps: both falling together is healthy, a widening gap is overfitting, a flat line is no learning, and a spike followed by nan is divergence. The stand-in below is a tiny diagnostician you can adapt to your own logs (trainer.state.log_history has the numbers).

Python Loss-curve doctor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Habits that prevent trouble

  • Pin versions (transformers, torch, datasets, tokenizers) in a lock file; many "it worked yesterday" bugs are upgrades.
  • Log everything: seed, git commit, data hash, arguments, metrics. A result you cannot reproduce is not a result.
  • Cache models and datasets on fast disk; set HF_HOME on machines with small home folders.
  • Start small: a tiny model and a 1,000-example subset run in minutes and expose most bugs.
  • Read the warnings. Truncation, "some weights were not used" and "token indices sequence length" warnings are usually telling you something true.
  • Trust but verify the metric: compute it by hand on five examples once.
  • Check licenses of both models and data before you ship (model cards).

Practice tasks

Work through these in a real environment (Colab or a local GPU). Each one reuses lessons from this course.

  1. Baseline versus Transformer. Pick a text-classification dataset from the Hub. Build a bag-of-words baseline, fine-tune DistilBERT, and report macro F1 for both on the same test split.
  2. Overfit test. Take 32 training examples and show your loop drives the loss near zero. Then break it on purpose (shuffle labels) and describe what changed.
  3. Memory budget. Using the memory math, predict the GPU memory for full fine-tuning of a 1B model, then for LoRA and QLoRA; run LoRA and compare.
  4. Metric tour. Evaluate one classifier with accuracy, precision, recall, F1 and a confusion matrix; find the two most confused classes and read 20 of the errors.
  5. Slices. Compute the metric by text length bucket and report the weakest slice with one hypothesis for why.
  6. Decoding experiment. Generate five completions with greedy decoding, beam search and sampling at three temperatures; write down how diversity and repetition change (decoding).
  7. Ship it. Push a model privately with a model card, then build a Gradio Space or a FastAPI endpoint with input limits and a health check.
  8. Embeddings. Index 200 paragraphs from your own notes, build a 30-query test set, and compare two embedding models by recall@5.

When you can do all eight without copying a recipe, you can move on to larger topics: multi-GPU training with Accelerate and DeepSpeed, preference tuning (DPO), long-context models, and retrieval-augmented generation in production.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A 3-class classifier starts training with a loss near 1.1. What does that suggest?

    Cross-entropy for uniform predictions over K classes is ln(K); ln(3) is about 1.10.
  2. 2. Why overfit a tiny batch first?

    If it cannot memorise 16 examples, the problem is data, labels, loss or LR, not data volume.
  3. 3. Training loss keeps falling while validation loss rises. This is...

    The model is memorising the training set; use early stopping or regularisation.
  4. 4. Which change most directly addresses a NaN loss in fp16 training?

    NaNs come from instability or overflow; bf16 has a wider range and warmup/lower LR stabilise training.

Summary

  • Debug in order: print a batch, run a tiny forward pass, overfit a handful of examples, check the initial loss, then scale up.
  • Match the symptom to the usual cause: memory, labels out of range, padding, NaN, flat loss, overfitting, leakage.
  • Read train and validation curves together; record seeds, versions and data hashes.
  • Practice end to end: baseline, fine-tune, evaluate, analyse errors, ship.

Next: Back to the Hugging Face Transformers track