Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Applications and Limits

Hallucinations, Bias and Safety

Understand why language models hallucinate, how bias and calibration problems appear, and how prompt injection works, with runnable consistency checks and a defensive prompt demo.

A language model is trained to produce plausible continuations, not true ones. Usually plausible and true overlap, which is why models are useful. When they do not, you get a hallucination: fluent, confident text that is wrong or unsupported. This lesson explains why that happens, how to detect and reduce it, and covers two related risks: bias and security (prompt injection).

Why models hallucinate

  • The objective rewards fluency. Next-token prediction has no "I do not know" signal unless training data or alignment puts one there.
  • Rare facts are weakly stored. A fact seen once in training is easy to blur. Questions about obscure people, papers or API functions are the classic trap.
  • Sampling adds risk. Higher temperature makes unlikely tokens more probable (decoding).
  • Missing context. Without the document in the prompt the model fills gaps from memory.
  • Pressure to answer. Models tuned to be helpful may guess instead of abstaining.
  • Compounding. Once a wrong token is emitted, every later token is conditioned on it.

Typical forms: invented citations, wrong numbers, fabricated function names, confident summaries of a document that was never provided, and "explaining" a false premise in the question.

Detecting hallucinations

No detector is perfect. Useful signals:

  1. Self-consistency. Ask the same question several times with sampling. Facts the model really "knows" repeat; guesses scatter.
  2. Grounding checks. Does every claim appear in the supplied source? (A model judge or string matching can test this.)
  3. Token probabilities. Very low probability on a key token (a name, a number) is a warning.
  4. External verification. Run the code, query the database, open the citation.

Below we simulate sampled answers (drawn from a seeded random generator, standing in for repeated model calls) and measure agreement.

Python Self-consistency as a confidence signal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

High agreement is not proof of truth (a model can be consistently wrong), but low agreement is a strong reason to distrust the answer or to abstain.

Calibration: does confidence match accuracy?

A model is calibrated if, among answers it gives with 80% confidence, about 80% are right. Instruction-tuned models are often overconfident. You measure it by binning answers by stated confidence and comparing with the observed accuracy. The gap, averaged over bins, is the expected calibration error (ECE).

Python Expected calibration error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each row shows stated confidence above actual accuracy. A calibrated system would have the two columns match.

Reducing hallucinations

  • Ground the model with retrieval and require citations.
  • Allow "I don't know" explicitly in the prompt and reward it in evaluation.
  • Lower temperature for factual tasks.
  • Use tools for arithmetic, dates and lookups (tool use).
  • Ask for verification steps or run a second pass that checks claims against the sources.
  • Constrain output formats so invalid values are rejected by code.
  • Measure it: keep a test set of questions with known answers and track the hallucination rate across prompt or model changes (harness).

Bias and fairness

Models learn from human text, so they absorb its stereotypes and gaps: associations between occupations and genders, uneven quality across languages and dialects, and under-representation of some regions. Bias shows up in generation, classification, ranking and in how a judge model scores answers.

Practical steps: audit outputs with paired prompts that differ only in a name or group, report quality per language and demographic slice instead of one average, document known limits, and involve people from affected groups in review. Removing sensitive words from a prompt does not remove bias that the model infers from other cues.

Prompt injection and other security risks

Prompt injection happens when untrusted text contains instructions and the model obeys them. It is the LLM version of SQL injection, except there is no clean boundary between "code" and "data": both are tokens.

Python Injection in a naive prompt and a hardened one

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Delimiters and keyword filters are speed bumps, not walls: a determined attacker can rephrase. Real defenses are structural:

  • Give the model least privilege: no tool that can send data out or delete things on its own.
  • Keep secrets (API keys, other users' data) out of the context window.
  • Separate untrusted content processing from privileged actions, and require human approval for risky steps.
  • Validate outputs (allowed URLs, schemas) before acting on them.
  • Log and monitor for abuse; rate limit.

Other risks: leaking training or context data, jailbreaks that talk a model out of its safety policy, harmful content generation, copyright and privacy concerns in training data, and over-reliance by users who treat fluent output as verified.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why can a model produce a confident but false statement?

    Next-token training rewards fluent continuations; nothing in the objective checks facts.
  2. 2. Sampling the same question 10 times gives 10 different answers. What does this suggest?

    Low agreement across samples is a classic sign of an unreliable answer.
  3. 3. A system states 90% confidence but is right 70% of the time. It is:

    Stated confidence exceeds actual accuracy, which is overconfidence.
  4. 4. Which is the strongest defense against prompt injection?

    Prompts can be overridden, so structural limits on privileges and data exposure matter most.

Summary

  • Hallucinations come from a fluency-first objective, weak memory of rare facts, sampling and missing context.
  • Detect them with self-consistency, grounding checks and external verification; measure calibration.
  • Reduce them with retrieval, abstention, lower temperature and tools.
  • Bias must be audited per slice; prompt injection is controlled by least privilege, not by wording.

Next: Project: An Evaluation Harness