Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 38 ยท Prompt Patterns in Production

Monitoring and Feedback Loops for Prompts

Watch a prompt after launch: structured logs, quality and cost metrics, user feedback, drift detection, alerts, and turning production failures into new test cases.

Passing your test set means the prompt worked on the cases you thought of. Production brings the cases you did not think of: new slang, a new product line, a document format nobody tested, a provider quietly updating a model. Monitoring tells you when quality, cost or safety moves; a feedback loop turns what you find into better prompts and bigger test sets.

This lesson covers what to log, what to measure, how to collect feedback and how to close the loop. The code below works on synthetic logs, so you can run every step.

What to log

For every model call, store a structured record (JSON, one per call), not just free text:

Field Use
request_id, timestamp, user_or_tenant (hashed) Tracing and per-user limits
prompt_name, prompt_version, model, settings Tie behaviour to the exact release
input_tokens, output_tokens, latency_ms, cost Budget and speed (cost and latency)
status / error / finish_reason Failures, timeouts, cut-off answers
guardrail_flags Which checks fired
feedback Thumbs, edits, escalations (added later)
input and output text Debugging, but see below

Text is the sensitive part. Redact it first (privacy), sample it (store 1 in 20 in full), restrict who can read it and expire it on a schedule. Metadata can be kept much longer.

Metrics worth watching

Group them in four families:

  1. Reliability: error rate, timeouts, retries, share of answers that fail format validation, share that hit max_tokens.
  2. Cost and speed: tokens per request, cost per request and per user, p50 and p95 latency (the slowest 5% matter: averages hide them).
  3. Quality signals: thumbs-down rate, regeneration rate, edit distance when users edit a draft, escalations to humans, an LLM judge score on a sample.
  4. Safety: guardrail trigger rate, canary hits, blocked-input rate.
Python Summaries from structured logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how the mean for version 1 (about 2,300 ms) hides that most calls were fast and one was terrible; the percentile and the invalid count point at it. Looking per prompt version is what lets you compare a release against its predecessor.

Collecting feedback

Feedback is the cheapest source of real labels. Collect it in layers, from lowest to highest effort for the user:

  • Implicit signals: the user regenerated, copied the answer, abandoned the chat, edited the draft heavily or contacted support afterwards.
  • Explicit signals: thumbs up/down, optionally with a reason picked from a short list ("wrong", "unhelpful", "unsafe", "too long").
  • Expert review: a staff member grades a random sample against a rubric. Slow, but the ground truth for calibrating cheaper signals.

Beware of bias: unhappy users click more than happy ones, and most users never click at all. Use feedback to find problems, not to compute absolute quality. A random sample reviewed by a human tells you the truth about the average.

Detecting drift

Quality can fall even when you change nothing:

  • Input drift: users start asking about new topics, in new languages or with longer inputs.
  • Model drift: the provider updates a model behind an alias.
  • Data drift: your knowledge base grows or goes stale, so retrieval returns different chunks.

The simplest detector compares a recent window of any metric with a baseline and alerts when it moves by more than normal noise.

Python Alert when a metric leaves its normal range

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Requiring several bad points in a row avoids paging someone for one noisy day. Tune the limit to your traffic: small volumes need wider limits.

Closing the loop

flowchart LR
    P["Production traffic"] --> L["Logs and feedback"]
    L --> F["Find failures: thumbs-down, guardrail hits, escalations"]
    F --> T["Add as test cases with correct labels"]
    T --> I["Improve the prompt"]
    I --> E["Run full test set"]
    E --> R["Release with canary"]
    R --> P

Every production failure becomes a permanent regression test.

The routine that makes prompts get better over time:

  1. Each week, pull a sample of bad outcomes (thumbs-down, invalid output, guardrail blocks, human escalations) plus a random sample of normal ones.
  2. A person reviews them, writes the correct answer or label and tags the failure type.
  3. Add them to the test set (anonymized).
  4. Fix the prompt (or retrieval, or the tool) and verify with a regression run.
  5. Release gradually, and watch the same metrics that exposed the problem.

Alerts and incidents

Decide in advance who is paged for what. Page on user-visible emergencies (error rate spike, safety flag spike, cost runaway); send quality drifts to a daily or weekly review instead. Keep a kill switch: a flag that turns off a feature, a tool or a prompt version immediately. Afterwards, write a short post-mortem and add a test so the same failure cannot return unnoticed.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why report p95 latency in addition to the average?

    Percentiles show the experience of the slowest users that the mean smooths over.
  2. 2. Which field lets you tie a bad answer to the exact prompt that produced it?

    Logging the prompt version and model makes any answer reproducible and comparable.
  3. 3. Why is thumbs-down feedback not a reliable measure of absolute quality?

    Use feedback to find problems; use random expert-reviewed samples to estimate real quality.
  4. 4. What is model drift?

    Aliases can point to updated models; pin versions and monitor quality to catch it.
  5. 5. What should happen to a production failure after it is analysed?

    Adding it to the test set ensures the same failure cannot silently return.

Summary

  • Log structured metadata for every call: prompt version, model, tokens, latency, status, flags, feedback; redact and sample the text.
  • Watch reliability, cost and speed, quality and safety metrics, per prompt version, with percentiles.
  • Collect implicit, explicit and expert feedback, remembering its bias; detect drift against a baseline.
  • Close the loop: failures become test cases, prompts improve, releases are gradual, kill switches exist.

Next: Project: A Support Ticket Classifier