Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 53 ยท Prompt Engineering

Reasoning: Chain of Thought and Beyond

Learn step-by-step prompting, scratchpads, self-consistency voting and decomposition, and when reasoning models make explicit chain of thought unnecessary.

A language model writes one token at a time, and each token can only use the computation of a single forward pass. A question like "what is 17 times 24 minus 9?" asks the model to do several dependent steps in one go. If it has to blurt out the answer immediately, it often gets it wrong. If it first writes out the intermediate steps, each step is easy and the final answer is far more likely to be right.

That is the idea behind chain-of-thought (CoT) prompting: let the model produce its working before the answer. This lesson covers the basic technique, how to combine several attempts with voting, how to break big problems into parts, and when modern reasoning models make all this unnecessary or even harmful.

Why writing steps helps

The text a model has already written becomes part of its input for the next token. Intermediate steps therefore act as a scratchpad: external memory that the model can read back. Writing "17 x 24 = 408" first makes "408 - 9 = 399" a trivial next step.

You can ask for this in two common ways:

  • Zero-shot CoT: add an instruction such as "Think through this step by step, then give the final answer on its own line."
  • Few-shot CoT: include examples (see Zero-Shot, One-Shot and Few-Shot Prompting) where the answer is preceded by worked reasoning, so the model copies the style.
TEXT
Question: A shop sells pens at 3 for $2. How much do 12 pens cost?
Think step by step, then finish with a line "Answer: <number>".

A good CoT prompt separates the reasoning from the final answer with a fixed marker. Your code can then read only the line starting with Answer: and ignore the rest. This is also a first taste of structured output.

Chain of thought in three moves

Ask for steps Tell the model to work before answering.
Mark the answer Use a fixed final line your code can parse.
Check the result Steps can be wrong. Verify what you can.

Visible steps help most on multi-step math, logic and planning.

1 / 2

Steps can still be wrong

CoT is not a guarantee. A model can write a fluent, confident chain with a mistake in step two, then carry the mistake to the end. The reasoning text is a generated explanation, not a window into a deterministic calculation. Two practical consequences:

  1. Whenever a tool can do the exact part (arithmetic, date math, lookups), let it. See Tool Use and Function Calling.
  2. Because sampling is random (see Logits, Softmax and Temperature), different runs produce different chains, and some are right while others are not. That variability can be turned into an advantage.

Self-consistency: vote over several chains

Self-consistency samples several independent reasoning chains with a temperature above zero, extracts each chain's final answer, and returns the most common one. Wrong chains tend to disagree with each other, while correct chains tend to converge on the same answer, so the majority is more reliable than any single sample.

The simulation below uses a fake model function. It is "right" 60% of the time and otherwise returns one of a few plausible wrong answers. We compare one sample against a majority vote of 9.

Python Majority vote beats a single sample

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy climbs as you add samples. The catch is cost: nine samples cost roughly nine times as much and take longer unless you run them in parallel. Self-consistency works best when answers are short and easy to compare (a number, a label, a choice). For long free-text answers there is no obvious "same answer" to vote on.

Voting only helps if errors are at least partly independent and the correct answer is the most likely single outcome. If the model is wrong more often than it is right on one specific answer, voting makes it more confidently wrong.

Decomposition: break the problem down

Some tasks are too large for a single chain. Decomposition splits the work into smaller prompts, each with a clear job, and passes the results forward. For example, to answer a question about a long report you might:

  1. Ask the model to list the sub-questions that must be answered.
  2. Answer each sub-question separately (possibly with retrieved text, see RAG).
  3. Combine the partial answers into a final response.

This costs more calls but gives you checkpoints: you can log, test and fix each step on its own, which makes debugging prompts far easier.

flowchart LR
    Q["big question"] --> S["split into<br/>sub-questions"]
    S --> A1["answer 1"]
    S --> A2["answer 2"]
    S --> A3["answer 3"]
    A1 --> C["combine"]
    A2 --> C
    A3 --> C
    C --> F["final answer"]

Decomposition turns one hard prompt into several easy ones with a combine step.

The next example shows the code side of CoT: a prompt builder and a parser that pulls out the marked final answer and tolerates messy output.

Python Parse the final answer from a reasoning chain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Returning None when the marker is missing matters: your program can then retry or flag the case instead of crashing or silently using garbage.

Reasoning models and when CoT hurts

Newer reasoning models are trained (often with reinforcement learning) to produce a long internal chain of thought before answering, and the API may bill those hidden "thinking" tokens as output. With these models:

  • You usually do not need "think step by step". The model already does it, and extra instructions can add noise or conflict with its own process.
  • Better prompts state the goal, the constraints and the output format clearly, then let the model plan.
  • You pay for thinking time, so a simple lookup or classification can become slower and costlier than necessary. Use a small, fast model for easy tasks.

Explicit CoT also hurts when:

  • the task is simple (spelling out a sentiment label gains nothing),
  • you need very short output or tight latency,
  • the model's reasoning leaks into places users see, or the "explanation" is invented after the fact and misleads people into trusting it.

A rule of thumb: use CoT or a reasoning model for multi-step problems with checkable answers, and skip it for plain extraction, classification and rewriting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does asking a model to write out its steps often improve answers to multi-step problems?

    Generated tokens are fed back as context, so intermediate results can be read by later tokens, making each step easier.
  2. 2. What does self-consistency do?

    Independent samples are extracted to final answers and the majority is chosen, since wrong chains tend to disagree.
  3. 3. When is explicit step-by-step prompting least useful?

    Easy single-step tasks gain nothing from reasoning text, which only adds tokens, cost and delay.
  4. 4. Why mark the final answer with a fixed prefix such as Answer:?

    A fixed marker lets your program extract just the result, and detect when it is missing.

Summary

  • Chain of thought lets a model use its own written steps as a scratchpad, which helps on multi-step problems.
  • Use a clear final-answer marker and parse it in code; handle the case where it is missing.
  • Reasoning text can contain mistakes; use tools for exact calculations.
  • Self-consistency samples several chains and takes a majority vote, trading cost for accuracy.
  • Decomposition splits a big task into checkable sub-prompts.
  • Reasoning models think internally, so they need clear goals rather than "think step by step", and simple tasks do not need them.

Next: Getting Structured Output (JSON).