Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 38 ยท Techniques: Few-shot, Chain-of-Thought and Roles

Choosing Good Few-Shot Examples

Pick few-shot examples that work: cover the label space, include edge cases, balance classes, watch ordering bias and select examples dynamically in Python.

Few-shot prompting is only as good as its examples. Three bad ones can make results worse than none. This lesson covers how to choose examples that teach the right lesson, how to avoid biases they can introduce, and how to pick them automatically.

Principles for good examples

1. Cover the label space. If there are four categories, show all four. A model rarely predicts a label it has never seen in the prompt.

2. Balance the labels. If five examples are "positive" and one is "negative", the model leans positive. Roughly equal counts reduce that bias, unless your real data is skewed and you want the lean.

3. Include hard and edge cases. Easy examples teach little. Include a borderline case with the correct answer so the model learns where the line is: sarcasm, mixed feelings, missing information, an input that fits no label.

4. Be diverse. Vary length, wording, topic and style. If every example starts "I love...", the model may latch onto that.

5. Make them correct. A wrong label in an example is a bug you are teaching. Review every example, ideally with someone else.

6. Match the real inputs. Use examples that look like what users will send, with the same typos, length and formatting.

Order and recency effects

Models can be sensitive to the order of examples. Common observations:

  • The last examples often have extra influence (recency).
  • A run of the same label at the end can bias the answer toward it.

Mitigations: shuffle examples, interleave labels, and test two or three orderings on your test set. If results swing a lot, your prompt is fragile; add clarity to the instruction or more examples.

Python Check label balance and coverage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The report shows two problems at once: "positive" dominates and "mixed" is missing entirely. Fix both before testing the prompt.

Interleaving to avoid runs

If the labels cluster, interleave them with a simple round-robin.

Python Interleave labels so no run of the same label ends the list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dynamic example selection

Instead of a fixed set, pick the examples most similar to the current input from a larger bank. This keeps prompts short while giving the model the most relevant guidance, and it is a standard production technique. Real systems measure similarity with embeddings; the stand-in below uses word overlap.

Python Pick the k most similar examples (word-overlap stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two caveats: dynamic selection can return all the same label (check balance after selecting), and it adds code to maintain. Use it when you have many examples and long inputs.

Examples can leak and mislead

  • Surface cues: if every "negative" example is short, the model may call short inputs negative. Vary length across labels.
  • Copying: models sometimes copy an example's wording or details into the answer. Use realistic but distinct content, and say "Do not reuse details from the examples".
  • Privacy: do not put real customer data in prompts you send to third parties.
  • Staleness: when the label definitions change, update the examples, or the prompt contradicts itself.

Explain, don't only show

Examples work best alongside a one-line rule. "Label 'mixed' when the review contains both praise and a complaint" plus an example is stronger than either alone. Some prompts add a short reason after each example label, which also teaches the decision logic (compare chain-of-thought).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should few-shot examples cover every label?

    Showing every label helps the model use the full label set.
  2. 2. Five examples are "positive" and one is "negative". What is the likely effect?

    Unbalanced examples bias the output toward the majority label.
  3. 3. What is a good way to reduce ordering bias?

    Ordering can matter. Interleaving and testing different orders reveals fragility.
  4. 4. What does dynamic example selection do?

    It keeps prompts short while supplying the most relevant guidance.
  5. 5. Which is true about wrong labels in examples?

    Examples are instructions; a wrong example is a bug.

Summary

  • Choose examples that cover all labels, are balanced, diverse, correct and realistic, and include edge cases.
  • Order matters: interleave labels and test orderings.
  • Select examples dynamically by similarity when you have a large bank, and re-check balance afterwards.
  • Pair examples with a short rule, and watch for surface-cue bias, copying, privacy and stale examples.

Next: Chain-of-Thought Prompting