Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 38 ยท Structured Output and Tool Use

Validating and Repairing Model JSON

Handle messy model output: extract JSON from chatter, fix common syntax slips, validate against a schema and retry with error feedback. Runnable Python.

Even with a good prompt, model output sometimes needs cleaning before it is usable: JSON wrapped in prose or code fences, trailing commas, single quotes, truncated objects. A robust pipeline treats the model like any unreliable external service: extract, parse, validate, and retry with specific feedback, never trusting the raw text.

flowchart LR
  A["Raw reply"] --> B["Extract JSON text"]
  B --> C["Parse"]
  C -->|"error"| R["Light repair"]
  R --> C
  C --> D["Validate schema"]
  D -->|"errors"| E["Retry with feedback<br/>(capped)"]
  E --> A
  D -->|"ok"| F["Use the data"]

A defensive pipeline: every stage can fail, and each failure has a defined response.

Step 1: extract the JSON text

Models often wrap output in markdown fences or add a sentence before it. Find the JSON by looking for a fenced block first, then the first balanced object.

Python Extract JSON from chatty replies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The extractor tracks string state so a } inside a string does not end the object early. A None result for an unbalanced object is a strong hint that output was truncated (check max tokens).

Step 2: parse, with light repair

Some slips are safe to fix automatically: trailing commas, Python-style True/None, single quotes. Be conservative: repairs that guess can silently change meaning.

Python Parse with safe repairs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Single-quoted keys (the last sample) are not fixed because a blind quote swap can corrupt text containing apostrophes. When a safe repair is not possible, fall back to a retry.

Step 3: validate

Parsing proves syntax only. Validate types, required keys, ranges and allowed values (see Getting Structured Output). Collect all problems rather than stopping at the first, so the retry message can mention every one.

Step 4: retry with feedback

When validation fails, send the model its own output together with the specific errors and ask for a corrected version. Specific feedback works much better than "try again". Cap the number of retries.

Python Retry loop with error feedback (stand-in model)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In production you would log each failure (input, reply, errors). A rising failure rate tells you the prompt, the schema or the model changed.

Design for failure

  • Default to a safe value (or route to a human) when retries run out, instead of crashing.
  • Never eval model output. Use a real JSON parser.
  • Keep outputs short enough to avoid truncation; unbalanced JSON usually means the cap was hit.
  • Prefer provider features that guarantee a schema when available, and keep your validation anyway.
  • Measure: track how often each stage (extract, repair, validate, retry) fires.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A JSON reply is unbalanced (missing closing braces). What is the likeliest cause?

    Truncation leaves objects unclosed, so raise the token cap or shorten the output.
  2. 2. Why be conservative with automatic JSON repairs?

    Safe fixes like trailing commas are fine; blind quote swaps can corrupt data.
  3. 3. What makes a retry message effective?

    Specific errors tell the model exactly what to fix.
  4. 4. Which should you never do with model output?

    eval executes arbitrary code; use a JSON parser instead.

Summary

  • Build a defensive pipeline: extract, parse (with conservative repairs), validate, retry with specific feedback, and cap retries.
  • Unbalanced JSON usually means truncation; check the max-tokens setting.
  • Collect all validation errors for the retry prompt; have a safe fallback when retries run out.
  • Never eval model output, and track how often each stage fires.

Next: Function Calling and Tool Use