Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 37 ยท Fine-tuning and Alignment

Supervised Fine-Tuning

Understand instruction tuning: chat templates, special tokens, loss masking on assistant tokens, packing, data quality and the hyperparameters that matter for SFT.

A base model trained only to predict the next token will happily continue your question with more questions. Supervised fine-tuning (SFT), also called instruction tuning, teaches it a conversation format: given a user message, produce a helpful assistant reply. The training signal is still next-token prediction, but the data is a set of curated (prompt, response) examples and the loss is computed only on the response.

In this lesson you build the pieces yourself: a chat template, a loss mask, and a quick look at how dataset quality and length affect training.

Instruction data

Each example is a list of messages with roles:

JSON
[
  {"role": "system", "content": "You answer in one sentence."},
  {"role": "user", "content": "What is a token?"},
  {"role": "assistant", "content": "A token is a small piece of text the model reads as one unit."}
]

Good SFT sets are small compared with pretraining (thousands to a few hundred thousand examples) but must be clean, diverse and consistent. Quality dominates: a few thousand careful examples often beat a million noisy ones. Sources include human-written answers, filtered model outputs (distillation) and rewritten existing datasets.

Chat templates

A model sees only token ids, so the roles must be flattened into text with special markers. Each model family has its own template, and you must use the template the model was trained with. If you format differently at inference time, quality drops.

Python Build a chat template

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At training time the whole conversation is shown. At inference time the template ends with the assistant marker so the model writes the answer, and generation stops when it emits <|end|>. Special tokens such as <|end|> are single vocabulary entries, not ordinary text. The track overview lists the lessons on how text becomes ids.

Loss masking

If we trained on every token, the model would also learn to write the user's messages. We usually want to learn only the assistant's answer. So we build a mask that is 1 on assistant tokens and 0 elsewhere, and average the cross-entropy loss over masked positions only.

Python Mask the loss to assistant tokens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The important line is the normalization: divide by the number of contributing tokens, not by the sequence length. Frameworks expose this as setting prompt labels to an ignore index (commonly -100), so the loss function skips them.

Packing and padding

GPUs want rectangular batches, but conversations have different lengths. Two strategies exist:

  • Padding fills short examples with a pad token up to the longest in the batch. Simple, but wasted compute.
  • Packing concatenates several examples into one sequence of fixed length, with attention masks that stop an example from attending to its neighbors. More efficient, but you must keep the masks correct.
Python Padding waste vs packing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The waste will differ from run to run of different data, but packing nearly always wastes much less.

Hyperparameters that matter

SFT is gentle compared with pretraining:

  • Learning rate is small, often 1e-5 to 2e-5 for full tuning (higher for LoRA, see LoRA and PEFT), with a short warmup and cosine decay.
  • Epochs: 1 to 3. More epochs on a small set memorize it. Watch validation loss, and stop when it turns up.
  • Batch size in tokens matters more than in examples; use gradient accumulation to reach it.
  • Sequence length must cover your longest realistic conversation, or the truncated tail teaches the wrong ending.

Data checks before training

  1. Deduplicate prompts, including near-duplicates.
  2. Remove examples that overlap your evaluation set (contamination).
  3. Read a random sample by hand. Wrong or rude answers will be imitated.
  4. Check the answer-length distribution: if every answer is long, the model will be wordy everywhere.
  5. Make sure the end-of-answer token is in the labels, or the model will never learn to stop.

What SFT cannot do

SFT shows the model one good answer per prompt. It cannot say that one acceptable answer is better than another, nor penalize a bad one. Preference methods handle that: see RLHF and DPO.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why mask the loss on prompt tokens in SFT?

    We want the model to learn the assistant's side of the conversation; masked positions are ignored in the average.
  2. 2. At inference time, how does the chat template end?

    The generation prompt opens the assistant turn; the model continues until it emits its end token.
  3. 3. What is the main benefit of sequence packing?

    Packing fills fixed-length sequences with several examples, but attention masks must still separate them.
  4. 4. Which is the safest statement about SFT dataset size?

    Quality and diversity dominate; noisy examples teach noisy behavior.

Summary

  • SFT continues next-token training on curated prompt-response pairs, formatted with the model's chat template.
  • The loss is averaged only over assistant tokens; include the end token so the model learns to stop.
  • Packing reduces padding waste; use low learning rates and few epochs.
  • Clean, deduplicated, uncontaminated data matters more than volume.

Next: LoRA and Parameter-Efficient Fine-Tuning