Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 53 ยท Prompt Engineering

Debugging and Improving Prompts

Debug prompts like code: classify failures, build test cases, fix ambiguity and instruction conflicts, weigh prompt length, and avoid common anti-patterns.

Sooner or later a prompt that worked in your five favorite examples fails on real traffic. The wrong response to "just tweak it until it looks right" is endless trial and error. The right response is the one you already use for software bugs: reproduce the failure, name its cause, change one thing, and re-run your tests.

This lesson gives you a taxonomy of prompt failures, a workflow built on test cases, and a list of anti-patterns to avoid.

A failure taxonomy

Naming the failure tells you which fix to try.

  • Ambiguity: the instruction has more than one reasonable reading ("summarize briefly": how brief?). Fix: be specific about length, audience and scope.
  • Missing context: the model lacks facts it needs, so it guesses. Fix: supply the facts, or retrieve them.
  • Instruction conflict: two rules disagree ("be extremely brief" and "explain every step"). The model picks one unpredictably. Fix: remove or rank the rules.
  • Format drift: correct content in the wrong shape, such as chatty text around JSON. Fix: stricter format rules, an example, or structured output.
  • Over- or under-triggering: a rule fires when it should not ("always apologize") or never fires. Fix: describe when the rule applies, and add a counter-example.
  • Reasoning errors: wrong logic or arithmetic. Fix: chain of thought, decomposition or a tool.
  • Hallucination: confident invention. Fix: grounding and an explicit "say you don't know" option (see Hallucinations).
  • Noise: the same input gives different outputs. Fix: lower temperature, tighter instructions, or accept and measure variation.

The debugging loop

Reproduce Save the failing input as a test case.
Diagnose Name the failure type.
Change one thing One edit per run, so you know what worked.
Re-run all tests Make sure you fixed more than you broke.
1 / 2

Work from test cases

Looking at a single output tells you almost nothing, because the same prompt may succeed on 7 of 10 inputs. Build a small test set: 10 to 30 representative inputs with the expected result, including awkward ones (empty input, very long input, another language, an attempted injection). Every time you find a new failure in production, add it.

Then compare prompt versions on the same set. The example uses a fake classifier whose behavior depends on what the prompt says, so the before/after difference is visible and deterministic.

Python Before and after on a test set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table makes each failure visible: version 1 had no neutral label, so it forced facts into positive or negative, and it misread "Not bad at all". Each fix in version 2 maps to one named failure. Always check that a fix did not break previously passing cases: that is a regression.

Ambiguity and conflicts

Read your prompt as if you were a very literal stranger with no context. Common traps:

  • Vague quantities: "a few examples", "short", "recent". Use numbers and dates.
  • Undefined terms: "important", "relevant", "professional tone". Say what you mean in one sentence.
  • Competing priorities: "be detailed" next to "keep it under 50 words". If two rules can clash, state which wins: "If these conflict, the word limit wins."
  • Pronoun confusion: "Compare it with the previous one" when the prompt contains three documents. Name things (<doc1>, <doc2>).

You can find many conflicts mechanically by listing every rule as its own line and checking pairs. The next example flags prompts with contradictory length rules and unresolved pronouns, a crude lint for obvious problems.

Python A tiny prompt linter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A linter like this is only a prompt for your own attention, not a verdict. The point is the habit: read the prompt critically, line by line.

Length trade-offs

Longer prompts are not automatically better.

  • More rules, more conflicts. Each added sentence is a new chance to contradict another.
  • Cost and latency grow with every token you send on every request (see Counting Tokens and Estimating Cost).
  • Attention is finite. Instructions buried in the middle of a long prompt are followed less reliably than those at the start or end (see Context Windows and the KV Cache).
  • Too short fails the other way: the model fills gaps with guesses.

A good approach: start with the shortest prompt that states the task, format and constraints, then add a rule only when a test case fails without it. When you add a rule, add the failing case to the test set. Occasionally try deleting a rule and see whether any test notices; if none does, the rule was dead weight.

Anti-patterns

  • Shouting and threats: "NEVER EVER do X!!!" Capital letters rarely help a modern model and make priorities muddy. Explain why or give the alternative to do instead.
  • Only negative rules: "Don't be long, don't use jargon, don't guess." Say what to do: "Answer in two sentences using everyday words."
  • Fixing by pile-up: adding a new sentence for every bug until nobody understands the prompt. Refactor instead.
  • Testing on one example: the fix may only work for the case you looked at.
  • Changing several things at once: you cannot tell which edit helped.
  • Hiding the format in prose: show an example output instead.
  • Trusting the model to grade itself without checks: see LLM-as-Judge and Human Review.

Keep a "bug log" of failures with the input, the bad output, the failure type and the fix. After a few weeks it shows you which kinds of failure your application suffers from most.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A prompt says 'be very brief' and 'explain every step in detail'. What kind of failure is this?

    Two rules disagree, so the model picks between them unpredictably. State which rule wins or remove one.
  2. 2. Why change only one thing per test run?

    With several simultaneous edits you cannot attribute the result to any of them.
  3. 3. What is a regression in prompt work?

    Re-running the whole test set after each change reveals cases that got worse.
  4. 4. Which rewrite of 'Don't be long' is better?

    A concrete positive instruction with a number is specific and testable.

Summary

  • Name the failure (ambiguity, missing context, conflict, format drift, reasoning, hallucination, noise) before changing the prompt.
  • Keep a test set of realistic and awkward inputs; add every new failure to it.
  • Change one thing at a time and re-run everything to catch regressions.
  • Replace vague words with numbers and state which rule wins in a conflict.
  • Prefer the shortest prompt that passes the tests; avoid shouting, negative-only rules and pile-ups.

Next: RAG: Giving Models Your Data.