Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 53 ยท Prompt Engineering

Getting Structured Output (JSON)

Make LLMs return reliable JSON: prompt for a schema, learn constrained decoding, validate with json and dataclasses, and retry when the output is invalid.

Chat answers are written for people. Programs need data: a ticket category, a list of extracted dates, a yes/no flag. If your code has to read a model's reply, you want a fixed shape such as JSON, not a friendly paragraph that says "Sure! Here is the data you asked for:".

This lesson shows how to ask for structured output, how providers can force it with constrained decoding, and how to validate and repair results in your own code. The golden rule: never trust model output until your code has checked it.

Asking for JSON in the prompt

The simplest approach is to describe the shape you want and show an example:

TEXT
Extract the order details from the message below.
Reply with ONLY a JSON object with these keys:
  "item": string
  "quantity": integer
  "urgent": boolean
No other text, no code fences.

Message: "Please send me 3 blue notebooks asap"

Good habits:

  • Name every key and its type. Say what to do when a value is missing (null, or an empty list).
  • Show one complete example of valid output (see few-shot prompting).
  • Say "no other text". Models still sometimes wrap JSON in Markdown fences or add a sentence before it, so your parser should cope.
  • Keep enums closed: "category is one of billing, bug, other".

Three layers of reliability

Prompt Describe keys, types and one example.
Constrain Let the API enforce a schema while decoding.
Validate Check the result in code and retry on failure.

Use all three for anything that feeds other software.

1 / 2

Schemas and constrained decoding

A JSON Schema is a machine-readable description of allowed JSON: which keys exist, their types, which are required, and allowed values.

JSON
{
  "type": "object",
  "properties": {
    "item": {"type": "string"},
    "quantity": {"type": "integer", "minimum": 1},
    "urgent": {"type": "boolean"}
  },
  "required": ["item", "quantity", "urgent"],
  "additionalProperties": false
}

Many APIs and local runtimes accept a schema and use constrained decoding (also called structured outputs or grammar-guided generation). Recall from Logits, Softmax and Temperature that the model scores every token in its vocabulary at each step. With constrained decoding the system looks at the schema, works out which tokens could still lead to valid JSON, and sets the scores of all other tokens to minus infinity before sampling. The model literally cannot emit a stray comma or a missing brace.

Two things constrained decoding does not do:

  • It guarantees the shape, not the truth. A perfectly valid {"quantity": 3} can still be the wrong number.
  • Complex schemas (deep nesting, many constraints) may be only partly supported, and a length cutoff can still truncate output mid-object. Always check the finish reason.

The toy below shows the idea at a tiny scale: a fake model proposes scores for tokens, and a mask removes the ones that are not allowed in this position.

Python Masking invalid tokens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without the mask, "maybe" is the single most likely token. With the mask, only the two valid tokens keep probability, and the model chooses between them.

Validating in your own code

Even with constrained decoding, validate. It protects you from provider differences, truncated output and your own schema mistakes. Python gives you two simple tools: json.loads for syntax and a dataclass (or a library such as Pydantic) for shape and types.

Python Parse, then validate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the explicit check that quantity is not a bool: in Python, True is an int, so a naive isinstance test would accept true as a quantity.

The retry-on-invalid loop

When validation fails, the cheapest fix is often to try again, this time showing the model what went wrong. A robust loop does this:

  1. Call the model.
  2. Parse and validate.
  3. If valid, return it.
  4. If invalid and attempts remain, append the bad output and the specific error to the conversation and ask for a corrected version.
  5. After a fixed maximum number of attempts, give up with a clear error (or a safe default).
flowchart LR
    P["prompt"] --> M["model call"]
    M --> V{"valid?"}
    V -- yes --> OK["return data"]
    V -- no --> R{"attempts left?"}
    R -- yes --> F["add error message<br/>to the prompt"]
    F --> M
    R -- no --> X["fail clearly"]

Validation feeds back into the prompt, with a hard cap on attempts.

The fake model below makes a mistake on its first two replies and succeeds on the third, so you can watch the loop work.

Python Retry loop with a fake model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real API code looks similar but calls the provider. The sketch below is read-only because the site cannot reach any model:

Python
# Read-only sketch: parameter names differ by provider
response = client.chat(
    model="some-model",
    messages=messages,
    response_format={"type": "json_schema", "json_schema": order_schema},
)
data = json.loads(response.text)   # still validate!

Practical tips

  • Keep schemas small. A few fields the model can fill reliably beat a 40-field monster.
  • Put reasoning first when you need it. If you ask for an answer key and a reasoning key, list reasoning first so the model thinks before committing (see Reasoning prompts).
  • Prefer enums and booleans to free text wherever the downstream code branches.
  • Log invalid outputs. They are your best test cases for debugging prompts.
  • Treat strings as untrusted. A valid JSON string can still contain an injection attempt; see Prompt Injection and LLM Security.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does constrained decoding guarantee?

    It blocks tokens that would break the grammar or schema, but it cannot check whether the values are correct.
  2. 2. Why validate output even when the API enforces a schema?

    Shape guarantees do not cover truncation, factual errors or logic your schema cannot express, so keep a validation step.
  3. 3. In a retry loop, what should you send back to the model after a failed attempt?

    Specific feedback tells the model what to fix, and a cap on attempts prevents endless loops.
  4. 4. Why check that a Python quantity value is not a bool?

    isinstance(True, int) is true, so true would slip through an integer check unless you exclude bools.

Summary

  • Ask for JSON by naming keys, types, missing-value rules and giving an example.
  • Constrained decoding masks invalid tokens so the shape is guaranteed, but not the truth.
  • Always parse and validate with json.loads plus type and range checks.
  • A retry loop sends the specific error back, with a hard attempt limit.
  • Small schemas, enums and logged failures keep structured output dependable.

Next: Prompt Templates and Versioning.