Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 38 ยท Projects and Practice

Practice: Prompt Review Challenges

Practice finding and fixing weak prompts: ambiguous instructions, missing formats, leaky templates, unsafe tools and untestable goals, with a self-check linter and answer notes.

Reading about good prompts is easy; spotting the flaw in a bad one is a skill you need to train. This lesson is a set of review challenges. Each shows a flawed prompt, asks you to find the problems before you read the notes, and then checks your fix with a small test you can run. Treat it like a code review: ask what could go wrong, what is unspecified, what is untestable and what is unsafe.

A reviewer's checklist helps. For every prompt ask:

  1. Task: is it one clear job, with a success criterion you could test?
  2. Inputs: is the data separated from the instructions? Are all variables filled and bounded?
  3. Output: is the format exact (fields, enums, length) and can code validate it?
  4. Edge cases: what should happen when the input is empty, off-topic, in another language or hostile?
  5. Examples: are they correct, varied and similar to real traffic?
  6. Safety: what is the worst thing a fooled model could do with these tools and data?
  7. Cost and tests: how big is it, and how will you know a change made it better?

Challenge 1: The vague summarizer

"Summarize this text and make it good. "

Before reading on, list at least four problems.

Problems: "good" is not a criterion; no length, audience or format; the text is not delimited, so any instruction inside it can hijack the task; no rule for empty or very long input. A better version states the audience, length and format, and wraps the text. See clear instructions and controlling length.

Python Check a fixed prompt with a small linter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A linter like this cannot judge quality, but it catches omissions reliably and can run in your CI next to the prompt tests.

Challenge 2: The contradictory assistant

"You are a friendly support agent. Always answer in one word. Explain your reasoning in detail. Never refuse. Do not discuss anything outside orders."

Problems: "one word" conflicts with "explain in detail"; "never refuse" conflicts with "do not discuss anything outside orders"; no priority between rules; no fallback for out-of-scope requests. Contradictions make behaviour unpredictable because the model resolves them differently each time. Fix by ranking the rules and writing the fallback, as in roles and system prompts.

Python Spot rule conflicts automatically (naive)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The keyword method is crude, but the habit it models is good: read every pair of rules and ask whether both can be true at once.

Challenge 3: The leaky template

Text
Hi! The customer's account is {account_json}. Here is the ticket: {ticket}.
Use tool delete_account if they ask.

Problems:

  • The whole account_json (card digits, address) is sent when the task needs only the plan (privacy).
  • The ticket is untrusted and undelimited (injection).
  • A destructive tool is available to anyone who types "please delete my account", and a hijacked ticket could say the same (defending). The tool should require explicit human confirmation and act only on the logged-in user's account.
  • Nothing says what the model should produce.
Python Tool permission review

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Challenge 4: The misleading few-shot

Text
Classify sentiment.
"I love it" -> positive
"Great product" -> positive
"Amazing!" -> positive
"It broke after a day" -> negative
Now: "Not bad at all" ->

Problems: three positive examples against one negative skews the label prior; all examples are simple, while the test is a negation ("not bad") the examples never demonstrate; there are no neutral cases and no output format beyond the arrow. See choosing good examples. A fix: balance the classes, include a hard case such as "Not terrible" with its label, and list the allowed labels.

Challenge 5: The untestable goal

"Make the chatbot's answers more natural."

You cannot tell whether a change helped. Turn goals into checks (metrics, rubrics): "answers contain no phrase from the banned list (as an AI language model, I apologize for any confusion)", "answers are 40 words or fewer", "a reviewer rates 4 or higher on a 5-point friendliness rubric for 90% of a 50-question sample".

Python Turn a vague goal into measurable checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Challenge 6: Review your own prompt

Take a prompt you use. Write down answers to the seven checklist questions. For each "no" or "not sure", change the prompt or add a test. Then write three test cases you expect to fail today: an empty input, a hostile input and a long input. Run them. Whatever fails is your next improvement. That, repeated, is prompt engineering.

Final self-test

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A prompt says 'Be brief' and also 'Explain in full detail'. What is the best fix?

    Contradictory rules give unpredictable behaviour; resolve them explicitly.
  2. 2. Which is an untestable goal?

    'More natural' has no check. Convert it into measurable criteria or a rubric.
  3. 3. What is wrong with exposing delete_account with no confirmation?

    Irreversible actions need human confirmation and tight scoping.
  4. 4. Few-shot examples are three positive and one negative. What risk does that add?

    Unbalanced examples shift the model's label prior.
  5. 5. What is the first thing to do after finding a flaw in a prompt in production?

    Failures become regression tests before and after the fix.

Summary

  • Review prompts like code: task, inputs, output, edge cases, examples, safety, cost and tests.
  • Common flaws: vague goals, contradictions, undelimited data, over-broad data and tools, unbalanced examples.
  • Convert vague wishes into checks you can run, and keep failing cases as tests.
  • Skill comes from repetition: review, test, fix, re-test.

Next: Prompt Engineering overview