Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 67 ยท CPU Design

Instruction Pipelining

Learn how a five-stage pipeline overlaps instructions, why it raises throughput but not latency, and how to compute speedup, with Python timing charts.

A single-cycle CPU finishes one instruction before it starts the next. Think of a laundromat with one customer at a time: wash, dry, fold, and only then does the next person begin. Everyone knows a better plan: when the first load moves to the dryer, start the second load in the washer. Each step is used all the time, and loads come out far faster, even though no single load got any quicker.

Pipelining is exactly this idea applied to instructions. The datapath is cut into stages, a new instruction enters every cycle, and several instructions are in different stages at once. In this lesson you will meet the classic five stages, learn why pipelining improves throughput but not latency, compute the speedup, and generate pipeline timing charts with Python.

The five stages

The datapath you studied already has five natural parts. Each becomes a pipeline stage:

Stage Name Work done
IF Instruction fetch Read the instruction at the PC; compute PC + 1
ID Instruction decode Decode, read the two source registers, sign-extend the immediate
EX Execute ALU operation, address calculation or branch comparison
MEM Memory access Load or store; other instructions pass through
WB Write back Write the result into the register file

Between stages sit pipeline registers (named after the stages they separate: IF/ID, ID/EX, EX/MEM, MEM/WB). At each clock edge, every register captures the results of the stage before it. This is what lets stage 3 work on one instruction while stage 2 already holds the next. Without them the signals of different instructions would mix.

flowchart LR
  IF["IF<br/>fetch"] --> R1["IF/ID"] --> ID["ID<br/>decode, read regs"] --> R2["ID/EX"] --> EX["EX<br/>ALU"] --> R3["EX/MEM"] --> MEM["MEM<br/>data memory"] --> R4["MEM/WB"] --> WB["WB<br/>write register"]

A five-stage pipeline: pipeline registers hold each instruction's values between stages.

Each stage takes one clock cycle. An instruction therefore needs five cycles from fetch to write back, but a new one can start every cycle.

The pipeline timing chart

Pipeline diagrams have time on the horizontal axis and instructions on the vertical axis. Each row is one instruction moving through the stages, each column one clock cycle. Reading down a column tells you what every stage is doing in that cycle.

The first TryIt draws the chart for six instructions and prints which instruction occupies each stage in every cycle. The instructions are chosen to be independent of each other, so no instruction needs a value that another has not produced yet. The next lesson, Pipeline Hazards, deals with what happens when they are not.

Python A pipeline timing chart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In cycle 5 the second table shows all five stages busy with instructions 5, 4, 3, 2 and 1. That is the pipeline when full. The first four cycles are the fill time, and the last four the drain, when stages sit empty. The 6 instructions finish in 10 cycles instead of 30, a cycle-count speedup of 3.00. With more instructions the ratio rises toward 5, because the fixed 4-cycle fill and drain matter less: for n instructions the formula is 5n / (n + 4).

Throughput versus latency

Two different things can be "fast":

  • Latency is how long one instruction takes from start to finish.
  • Throughput is how many instructions finish per unit time.

Pipelining improves throughput and does not improve latency. In fact it makes latency a little worse, for two reasons. Every stage must be as long as the slowest stage (the clock is the same for all), and the pipeline registers add a small delay to every stage.

The next TryIt uses the stage delays from the datapath lesson plus 20 ps of register overhead per stage, and compares the single-cycle machine with the pipelined machine.

Python Latency, throughput and speedup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a single instruction, the pipelined machine is actually slower: 1050 ps against 660 ps. At 10,000 instructions the speedup has climbed to about 3.1 and it levels off at 3.14, not at 5. The reason is that the stages are unbalanced: the slowest one (MEM, 190 ps) sets the clock, while WB needs only 80 ps. With perfectly balanced stages the speedup would be 4.34, still below 5 because of register overhead. Ideal speedup equals the number of stages only if the stages are balanced, overhead is zero and the pipeline never stalls.

Users feel throughput on long workloads such as video encoding or compiling, because they run billions of instructions. Latency matters for a single tiny task, where the pipeline never gets full. This is also why a CPU's clock speed alone says little about its speed. See Measuring Performance.

Why RISC pipelines well

The design leans on properties that RISC instruction sets deliberately have:

  1. Fixed-length instructions. Fetch always reads one word, and decode can start immediately. With variable-length instructions (x86), even finding where the next instruction begins takes work. Modern x86 chips solve this by translating to internal fixed-size micro-operations first.
  2. Few instruction formats and the source registers in the same bit positions. The ID stage can read the register file before it fully knows the instruction.
  3. Memory is touched only by loads and stores. Only one stage (MEM) does it, and only the ALU address calculation comes before it.
  4. Aligned memory accesses that take one cycle in the best case.

Those points are why Instruction Set Architecture choices affect hardware performance so strongly.

More stages: diminishing returns

If five stages are good, are fifty better? Cutting the logic into more, shorter stages shortens the clock, but each stage still pays the register overhead, and hazards cost more cycles each time (a deeper pipeline has more instructions in flight to throw away after a mistake). The final TryIt explores the first effect: 660 ps of total logic is divided evenly into k stages, with 20 ps of overhead each. It prints a table and draws a plot of the speedup over a single-cycle design.

Python Pipeline depth and diminishing returns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one stage the "pipeline" is even slower than the single-cycle machine (0.97), because it pays the register overhead for nothing. Five stages give 4.34, twenty give 12.45 and fifty reach 19.88, well below both the ideal line and the absolute limit of 33 (660 / 20). The curve flattens. Real processors confirm this: Intel's Pentium 4 pushed to about 20-31 stages for high clock speeds and paid for it with heat and long penalties after mispredicted branches. Modern high-performance cores use roughly 12-20 stages, and their performance comes from other techniques, like those in Superscalar, Out-of-Order and Speculative Execution.

Pipelining and the performance equation

In the equation time = instructions x CPI x clock period, pipelining attacks the clock period while keeping CPI near 1 (see Measuring Performance). Real CPI is a bit above 1 because hazards force the pipeline to wait. That bookkeeping is the subject of the next lesson.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How many cycles does an ideal 5-stage pipeline need for 20 instructions?

    It takes 5 cycles to fill and then finishes one instruction per cycle: 5 + 20 - 1 = 24 cycles.
  2. 2. What does pipelining improve?

    More instructions finish per second, but each one still takes at least as long (usually slightly longer) from start to end.
  3. 3. What is the job of the pipeline registers?

    At each clock edge a pipeline register captures the outputs of the previous stage, so stage inputs stay stable for one cycle.
  4. 4. Why is the ideal speedup of a 5-stage pipeline less than 5 in practice?

    The clock is set by the slowest stage plus register overhead, and hazards add stall cycles, so the speedup falls short of the number of stages.
  5. 5. Which property helps pipelining most in RISC instruction sets?

    Fixed-length, regular instructions make fetch and decode simple and quick, so each stage does a small, predictable amount of work.

Summary

  • A pipeline splits instruction execution into stages (IF, ID, EX, MEM, WB) separated by pipeline registers, so several instructions are in progress at once.
  • A full pipeline completes one instruction per cycle; n instructions need stages + n - 1 cycles.
  • Pipelining raises throughput; the latency of a single instruction does not drop and may rise.
  • Speedup is limited by unbalanced stages, register overhead, the fill and drain time, and stalls.
  • Making a pipeline deeper gives diminishing returns and makes mistakes more expensive.
  • RISC instruction sets (fixed length, load/store) are designed to pipeline well.

Next, see what goes wrong when instructions depend on each other: Pipeline Hazards.