Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 67 ยท Parallel Architectures

SIMD, Vector Processing and GPUs

Understand vector instructions, why NumPy feels like SIMD, how a GPU organizes threads into warps, and the difference between latency and throughput.

Much of the work computers do is repetitive: brighten every pixel, multiply every weight in a neural network, add two columns of numbers. The same operation is applied to thousands or millions of values. Hardware that exploits this is called data-parallel, and it is the reason a modern graphics card can do trillions of arithmetic operations per second.

In this lesson you will see how a CPU's vector instructions process several values per instruction, why array libraries such as NumPy are so much faster than Python loops, how a GPU organizes thousands of threads, and why GPUs trade single-task speed for overall throughput. It follows Parallel Processing and Flynn's Taxonomy, where SIMD was one of the four categories.

SIMD: one instruction, many values

A normal (scalar) add instruction takes two numbers and produces one. A SIMD add instruction works on a vector register that holds several numbers side by side, called lanes, and adds all lanes at once.

flowchart LR
  subgraph A["Register A"]
    a0["1"] --- a1["2"] --- a2["3"] --- a3["4"]
  end
  subgraph B["Register B"]
    b0["10"] --- b1["20"] --- b2["30"] --- b3["40"]
  end
  subgraph R["Result (one instruction)"]
    r0["11"] --- r1["22"] --- r2["33"] --- r3["44"]
  end
  A --> R
  B --> R

One vector add produces four results.

The width of the register determines the number of lanes. A 128-bit register holds four 32-bit floats (SSE on x86, NEON on ARM); a 256-bit register holds eight (AVX2); a 512-bit register holds sixteen (AVX-512). Vector instruction sets also include lane-wise multiply, compare, minimum and maximum, shuffles that rearrange lanes, and masks that switch individual lanes off (so that if statements can be applied to some lanes only). RISC-V's vector extension and ARM's SVE go further: the code does not hardcode the width, and the hardware tells the program how many lanes it has.

A loop over N elements becomes N divided by the lane count iterations, plus a short tail for the leftover elements that do not fill a whole vector. The simulation below counts instructions for a plain loop and for different vector widths.

Python Counting vector instructions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the speedup stays under the lane count, and the gap grows with the width (13.7x instead of 16x), because of the scalar tail. Real compilers produce this structure automatically in a step called auto-vectorization, or you can ask for specific vector instructions with compiler intrinsics. Vectorization works best when the data is contiguous in memory, aligned, and the loop iterations do not depend on each other. A loop like a[i] = a[i-1] + 1 cannot be vectorized naively, because each result needs the previous one.

NumPy as a SIMD analogy

When you write a + b for two NumPy arrays, no Python loop runs. NumPy calls a compiled C routine that loops over a contiguous block of memory, and that routine uses SIMD instructions where the CPU supports them. Two things make this fast: the loop runs in compiled code instead of the interpreter, and the data is tightly packed numbers rather than Python objects scattered across memory. (In the in-browser runner, NumPy runs as WebAssembly, so the exact speed differs from your desktop, but the contrast between loops and arrays remains.)

Python A Python loop against a NumPy array operation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two lines are always the same; the speedup varies by machine and run, but is typically tens to hundreds of times. Not all of it is SIMD (most comes from avoiding interpreter overhead), but the habit it teaches is the same one that drives vector hardware: describe the operation on the whole array, and let the machine do the repetition.

Vector processors

Vector hardware is older than most people think. Supercomputers such as the Cray-1 (1976) had vector processors with long vector registers and pipelined units that streamed through elements. Modern SIMD units are a descendant of that idea, and the newer variable-length vector extensions (ARM SVE, RISC-V V) return to it.

GPUs: thousands of simple lanes

A GPU (graphics processing unit) was designed to compute colors for millions of pixels, each independently. That job is embarrassingly parallel, so GPU designers spent their transistors on many simple arithmetic units instead of on large caches and clever branch predictors. The result is a chip with thousands of arithmetic lanes, which also turned out to be ideal for matrix multiplication, the heart of machine learning.

The programming model (CUDA from NVIDIA, HIP, OpenCL, and similar) is built around a kernel: a small function that you launch over a large grid of threads. Every thread runs the same code, and uses its index to decide which data element to handle. Threads are organized hierarchically:

  • A thread handles one element.
  • Threads are grouped into blocks (also called work-groups). Threads in a block can share fast on-chip memory and synchronize.
  • Blocks form a grid covering the whole problem.
  • The hardware executes threads in groups of 32 (NVIDIA calls it a warp; AMD uses 32 or 64 and calls it a wavefront). All threads of a warp run the same instruction at the same time, which is SIMD in disguise, sometimes called SIMT (single instruction, multiple threads).
flowchart TB
  G["Grid (the whole problem)"] --> B0["Block 0"]
  G --> B1["Block 1"]
  G --> B2["Block 2"]
  B1 --> W0["Warp 0: threads 0-31"]
  B1 --> W1["Warp 1: threads 32-63"]
  W0 --> T["Each thread has its own index"]

A kernel launch is a grid of blocks, each made of warps of 32 threads.

A thread finds its element with a formula: global_id = block_index * block_size + thread_index. The example below reproduces that mapping and also demonstrates warp divergence. If threads within a warp take different sides of an if, the hardware runs both paths one after the other, turning off the lanes that do not apply. So divergence costs time, while an if whose outcome is the same for the whole warp is cheap.

Python Thread indexing and warp divergence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first condition makes every warp run both branches (3 cycles for each of 4 warps, 12 in total). In the second, entire warps agree, so each executes just one side (2 cycles per warp, 8 in total). Organizing data so that neighboring threads take the same path is a core GPU optimization.

GPUs also depend on memory behavior. Neighboring threads should read neighboring addresses, so the hardware can merge their requests into one wide transaction (coalescing). Each block also gets a small, fast shared memory that works like a programmer-managed cache. And to hide the long delay of main memory, a GPU keeps many warps in flight: when one warp waits for data, the scheduler instantly switches to another, with no context-switch cost.

Latency versus throughput

This leads to the key design contrast. A CPU core is a latency-optimized machine: big caches, deep branch prediction, out-of-order execution, all aimed at finishing one task as quickly as possible. A GPU is throughput-optimized: each thread is slow, but there are so many that the total work per second is huge. Latency is the time for one job; throughput is the number of jobs per second.

GPUs also pay a fixed cost to start: data must be copied over PCIe (see Buses and Interconnects) and the kernel launched. For small jobs this overhead dominates, and the CPU wins. For large jobs the GPU's raw width wins. The toy model below uses made-up but plausible numbers to show the crossover.

Python When does the GPU pay off?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers are invented for illustration, but the shape is real: tiny jobs belong on the CPU, big regular jobs belong on the GPU, and the best programs keep data on the GPU for as long as possible to avoid paying the copy cost again and again.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A 256-bit vector register holds 32-bit floats. How many values does one vector add process?

    256 bits divided by 32 bits per float gives 8 lanes.
  2. 2. Why is a NumPy array addition much faster than a Python for loop over lists?

    The loop executes in compiled code over tightly packed data, avoiding interpreter overhead and object indirection.
  3. 3. On a GPU, what is a warp?

    Threads are scheduled in groups; all threads of a warp execute the same instruction at once, which is why divergence costs time.
  4. 4. Threads in one warp take different sides of an if statement. What happens?

    SIMT hardware serializes divergent paths, so the warp takes the time of both branches.
  5. 5. Which statement about CPUs and GPUs best captures the latency/throughput difference?

    A CPU core is built to finish one job quickly. A GPU runs enormous numbers of slower threads to get high total throughput.

Summary

  • SIMD instructions apply one operation to every lane of a vector register; wider registers mean more lanes. Loops end with a scalar tail.
  • Vectorization needs independent iterations and contiguous data; compilers can do it automatically, and array libraries like NumPy give you the same effect.
  • GPUs run a kernel over a grid of blocks of threads; threads execute in warps of about 32 that share instructions (SIMT).
  • Divergent branches inside a warp run both paths; coalesced memory access and shared memory are key GPU optimizations.
  • CPUs minimize latency, GPUs maximize throughput, and the copy and launch overhead means small jobs stay on the CPU.

Continue with Power, Energy and Thermal Limits.