Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 53 ยท Customizing Models

Fine-Tuning vs Prompting

Decide between prompting, RAG and fine-tuning, build and validate a chat JSONL dataset, and learn how to evaluate, cost and avoid overfitting.

Sooner or later someone says, "Let's fine-tune the model." Sometimes that is exactly right, but more often a better prompt, a few examples or retrieval would solve the problem in a day instead of a month. Fine-tuning means continuing to train an existing model on your own examples so that its weights change. Prompting changes only the input and leaves the model untouched.

This lesson helps you choose, shows the data format most fine-tuning services expect, and gives you a validator to catch bad data before you pay for training. The training run itself cannot happen in the browser, so that part is read-only.

The ladder: try the cheap things first

Climb these rungs in order and stop as soon as quality is good enough.

Cheapest fix first

Better prompt Clear task, format and constraints.
Few-shot examples Show the pattern in the prompt.
RAG Fetch the facts at question time.
Fine-tune Change the model itself.

Each step costs more time, data and maintenance than the last.

1 / 2

Refer back to Anatomy of a Good Prompt, Few-Shot Prompting and RAG for the first three rungs.

What fine-tuning is good at

Fine-tuning teaches behavior, not facts. It works well for:

  • A consistent style or format: always answer in your brand voice, or always emit a specific JSON shape.
  • A narrow, repeated task: classify tickets into your 40 internal categories, extract fields from your document type.
  • Shorter prompts: behavior moves into the weights, so you stop paying for long instructions and many examples on every call.
  • Smaller, cheaper models: a small model tuned on one task can match a large general one on that task.
  • Hard-to-describe skills: a tone or domain jargon that is easier to show than to explain.

It works poorly for:

  • Fresh or changing facts. A tuned model still forgets and still makes things up. Use RAG, which can cite sources and updates in minutes.
  • Private knowledge lookups. The model cannot reliably recall one fact from a few thousand examples.
  • Fixing a vague task. If humans disagree on the right answer, training data will be noisy and so will the model.
Need Best first choice
Answer from company documents RAG
Output in a strict JSON shape Prompt with schema, then fine-tune if it still slips
Consistent tone across thousands of replies Fine-tune (after trying few-shot)
Up-to-date facts RAG or tools
Cut cost on a high-volume narrow task Fine-tune a smaller model

The data format: chat JSONL

Most providers and open-source trainers accept JSONL: one JSON object per line, each a complete example. For chat models, an example is a list of messages, the same roles you saw in Calling an LLM API. The last assistant message is what the model learns to produce.

JSON
{"messages": [{"role": "system", "content": "You write short, friendly product replies."}, {"role": "user", "content": "Where is my order?"}, {"role": "assistant", "content": "Thanks for asking! Open Orders in your account to see live tracking."}]}

Details vary between providers, so check the current documentation for yours. Two other formats you will see are instruction/response pairs ({"prompt": ..., "completion": ...}) and preference pairs for alignment.

Data quality beats data quantity

A few hundred excellent, diverse examples usually beat thousands of sloppy ones. The model copies what you show it, including your mistakes.

  • Consistency. If two annotators answer the same question differently, the model learns to be erratic.
  • Coverage. Include easy, hard, short, long and edge cases, plus examples where the correct behavior is to refuse or say "I don't know".
  • Balance. If 95% of labels are one class, the model will lean toward it.
  • No leaks. Remove personal data (see Bias, Privacy and Copyright) and anything you would not want repeated.
  • Hold out a test set before you start. Never train on it.

A dataset builder and validator

Cheap checks catch most problems. The next example builds JSONL from raw records and then validates structure, roles, empty answers, duplicates and label balance.

Python Build a chat JSONL dataset and validate it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The validator found a duplicate and an empty answer, and the label counts show how many examples each class has. Real validators also check token length, banned content and train/test overlap.

Evaluating before and after

Never judge a tuned model by feeling. Measure it against the baseline you already have: the best prompt on the base model.

  1. Build a held-out test set that is never used for training.
  2. Score the baseline (prompted base model) with your eval harness.
  3. Fine-tune, then score the tuned model on the same set with the same metric.
  4. Compare on slices (rare classes, long inputs), and check for regressions on general tasks.
  5. Keep the prompt short for the tuned model, and compare cost per request as well as quality.
Python Compare baseline and tuned scores with fake models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Overfitting and forgetting

Training minimizes error on the training set, so a model can start to memorize it instead of learning the pattern. Signs: training loss keeps dropping while validation loss rises, answers repeat training sentences, and the model gets worse on inputs unlike your data. Remedies: fewer epochs (1 to 3 is common), a lower learning rate, more varied data, and early stopping when validation loss stops improving. Catastrophic forgetting is the related loss of general skills; techniques such as LoRA limit it by changing only a small part of the model.

Costs and maintenance

  • Training cost: hosted fine-tuning bills per training token times epochs; open-weight tuning bills GPU hours. Both are usually small compared with the engineering time to prepare data.
  • Serving cost: a tuned hosted model can cost more per token than the base model, and you may need to run your own endpoint for open-weight models.
  • Staleness: every new base model release means re-tuning, so keep your dataset and eval set in version control.
  • Ownership: you must maintain, monitor and re-evaluate the model like any other component.

Read-only sketch of what a hosted job looks like (names differ per provider):

Python
# Not runnable here: needs an account and network access.
client = SomeProviderClient()
file = client.files.upload("train.jsonl", purpose="fine-tune")
job = client.fine_tuning.create(training_file=file.id, model="base-model", epochs=2)
print(job.status)

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Your chatbot must answer from a policy handbook that changes weekly. What should you try first?

    Fine-tuning teaches behavior, not reliable facts, and retraining weekly is slow. RAG fetches current text and can cite it.
  2. 2. Which belongs in a chat fine-tuning example?

    Each example shows a conversation whose final assistant message is the target output.
  3. 3. Validation loss rises while training loss keeps falling. What is likely happening?

    A growing gap between training and validation loss is the classic sign of memorizing training data instead of generalizing.
  4. 4. Why hold out a test set before training?

    Only data the model never trained on gives an honest comparison.

Summary

  • Try prompts, few-shot examples and RAG before fine-tuning; fine-tune for style, format, narrow tasks and cheaper models.
  • Fine-tuning teaches behavior, not up-to-date facts.
  • Chat JSONL holds one conversation per line; validate structure, empties, duplicates and balance before training.
  • Compare tuned and baseline models on a held-out test set, and watch for overfitting and forgetting.
  • Count maintenance as part of the cost.

Next: LoRA, Quantization and Efficient Models.