Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 31 ยท Why GPUs

What Is GPU Computing

Learn what GPU computing is, how a GPU differs from a CPU, which workloads fit data parallelism, how CUDA started, and when a GPU is the wrong tool.

A graphics processing unit (GPU) began life as a chip that drew triangles and colored pixels. Drawing a frame means doing the same small calculation for millions of pixels, so GPU makers built chips with thousands of simple arithmetic units working side by side. Around 2006, NVIDIA released CUDA, a way to write ordinary-looking C++ programs that run on those units. Since then the GPU has become the engine behind deep learning, scientific simulation, video processing, ray tracing and much more.

This tutorial teaches CUDA from the ground up: how to think about thousands of threads, how memory works, how to make code fast, and how to avoid the classic bugs. This first lesson gives you the mental model. You do not need any GPU knowledge, but you should be comfortable with C or C++ basics (see the C tutorial or C++ tutorial if you need them).

Two chips, two goals

A CPU and a GPU are both processors, but they are built to optimize different things.

  • A CPU is designed to finish one task as fast as possible. It spends most of its silicon on large caches, branch prediction and out-of-order machinery, so a single thread of ordinary code runs quickly. A desktop CPU has somewhere between 4 and 64 cores.
  • A GPU is designed to finish a huge number of similar tasks per second. It spends its silicon on arithmetic units and gives each one very little control logic. A modern GPU has thousands of arithmetic lanes grouped into dozens or hundreds of larger blocks.
flowchart LR
    subgraph CPU["CPU: few strong cores"]
        C1["Core<br/>big cache"]
        C2["Core<br/>big cache"]
        C3["Core<br/>big cache"]
        C4["Core<br/>big cache"]
    end
    subgraph GPU["GPU: many simple lanes"]
        G1["lane lane lane lane"]
        G2["lane lane lane lane"]
        G3["lane lane lane lane"]
        G4["lane lane lane lane"]
    end

Schematic only: a CPU invests in few powerful cores, a GPU in a very large number of small ones.

Latency versus throughput

Two words capture the difference.

Latency is how long one task takes from start to finish. Throughput is how many tasks finish per second. A sports car has low latency for one passenger; a train has high throughput for a thousand. The CPU is the sports car, the GPU is the train.

A GPU hides the slowness of individual operations (such as a memory read that takes hundreds of cycles) not with caches but with sheer numbers: while some threads wait for data, the hardware switches to other threads that are ready. This only works when you give it far more threads than it has lanes, which is why GPU programs launch thousands or millions of threads.

The next example is a small model, not a measurement. It compares a pretend CPU with 8 fast workers to a pretend GPU with 2,048 slower workers, each doing one unit of work at a time. The numbers are invented to show the shape of the trade-off.

Python A toy throughput model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one task the CPU wins easily because the GPU worker is slower. With millions of independent tasks the GPU wins by a wide margin. The crossover is the whole story of GPU computing: a GPU pays off only when there is a lot of parallel work. A real GPU also pays a fixed cost to move data and start work, which pushes the crossover even higher (you will model that in Moving Data).

Data parallelism

The kind of parallelism GPUs love is data parallelism: the same operation applied to many pieces of data, where each piece can be processed without waiting for the others.

Brightening a photo is the classic example. Every output pixel depends only on the matching input pixel, so you could process the pixels in any order, or all at once, and get the same picture. Adding two arrays element by element, applying an activation function to a million neurons, or testing a million candidate passwords are all the same shape.

You can check the "any order" property directly. The following snippet processes an array in a random order and compares it with the normal order.

Python Independent work can run in any order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both results match because no pixel needs another pixel's result. When work does depend on other work (a running total, a chain of steps), you need special patterns such as the ones in Parallel Reduction and Scan and Histogram.

Why sequential speed stopped growing

For decades programmers got faster programs for free. Moore's law (the observation that the number of transistors on a chip doubles roughly every two years) went hand in hand with rising clock frequency, so the same single-threaded code ran faster on every new CPU. Around the mid-2000s that free lunch ended, for three physical reasons.

  • The power wall. A chip's dynamic power grows with its clock frequency and with the square of its voltage, and higher clocks need higher voltage. In practice, doubling the frequency of one core costs far more than double the power, and all of it leaves as heat. Chips hit the limit of what a package and a cooler can remove.
  • The memory wall. Processors got faster much more quickly than main memory did, so a core often sits idle waiting for data. Adding clock speed to a core that is already waiting changes little.
  • The instruction-level wall. CPUs already squeeze parallelism out of a single instruction stream. A pipeline works like an assembly line: while one instruction executes, the next is decoded and the one after that is fetched (the classic stages are fetch, decode, execute, write-back). Out-of-order execution and branch prediction keep the line full. Hyper-threading (hardware multithreading) lets one physical core hold two instruction streams and switch to the other when the first stalls. All of this helps, but each extra trick yields less than the one before.

The industry's answer was to stop making one core faster and instead put many cores on a chip. That only helps if the program has work that can run at the same time. The data explosion (photos, video, sensor logs, training sets) supplies exactly that kind of work, and moving data and computing near it is also more energy efficient than racing a single core at a high clock.

A tiny model shows the trade-off. Power is modeled as cores x frequency cubed (voltage rises with frequency). Speed follows Amdahl's law: the serial part of a job runs on one core, the parallel part is shared by all cores.

Python One fast core or many slower cores?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table. Doubling one core's clock costs 8x the power for 2x the speed. Eight cores at half the clock use the same power as one core at full speed and are four times faster on fully parallel work, but slower when half the job is serial. Parallel hardware wins only on parallel work, which is why this tutorial keeps asking whether your problem is data parallel.

How a GPU job flows

A GPU program is an example of heterogeneous computing: two different kinds of processor, each with its own memory, cooperate. The CPU is the host and the GPU is the device. A typical run follows the same steps every time.

  1. Prepare. The host allocates memory on both sides and fills its own arrays with input data.
  2. Offload. The host copies the input from host memory to device memory over the PCIe or NVLink connection.
  3. Execute. The host launches a kernel. Thousands of device threads run the same function on different pieces of the data.
  4. Retrieve. The host copies the results back and uses them, then frees the memory.

The copies in steps 2 and 4 are not free, which is why Host-Device Transfers and Streams and Events matter later.

Which workloads fit

Good fits share three traits: lots of independent work, plenty of arithmetic per byte moved, and regular memory access.

  • Matrix and tensor math, which powers neural-network training and inference.
  • Image, video and signal processing: filters, resizing, encoding, FFTs.
  • Physics and scientific simulation: n-body problems, fluid dynamics, molecular dynamics.
  • Monte Carlo methods and finance: millions of independent random trials.
  • Graphics and ray tracing, the original job.
  • Large-scale search, sorting and database scans over columns of data.
  • Autonomous systems: cameras and lidar must be turned into decisions in milliseconds.
  • Cryptography-style workloads such as hashing and mining, where the same function is tried on enormous numbers of inputs.
  • Scientific research such as weather, genomics and protein folding, and deep learning in general.

When not to use a GPU

A GPU is not a faster CPU. It is the wrong choice when:

  • The problem is small. Copying data to the GPU and launching work has a fixed cost that a tiny job never earns back.
  • The work is mostly sequential, with each step needing the previous result (parsing text, many recursive algorithms).
  • There is a lot of branching with different threads wanting different paths (warp divergence wastes lanes).
  • The memory access is random and irregular, such as chasing pointers through a linked list.
  • A library already does it. For matrix multiply, FFTs or sorting, calling a tuned library beats writing your own kernel (The CUDA Library Ecosystem).

Always keep a CPU version as the baseline. Speed-up claims only mean something against a well-optimized CPU implementation.

A short history

Early GPUs had fixed-function pipelines. In the early 2000s researchers noticed that programmable shaders could be bent into doing general math, a trick called GPGPU (general-purpose computing on GPUs), but it meant disguising your data as textures. NVIDIA introduced CUDA in 2006-2007 so programmers could write kernels in a C-like language with explicit control over memory and threads. Other vendors followed with OpenCL, and later AMD's HIP/ROCm and the SYCL standard. CUDA remains the most widely used because of its tooling and its libraries (cuBLAS, cuDNN, Thrust and more); Multi-GPU, Portability and the Wider Landscape compares the options.

What runs on this site

Be clear about what you are getting. The site cannot run CUDA: there is no NVIDIA GPU, no nvcc compiler, and your browser tab is not a CUDA device. So every lesson works like this:

  • Real CUDA C++ appears in plain, read-only code blocks. This is genuine code you can paste into a .cu file on a machine with a GPU.
  • Runnable examples are stand-ins written in Python. They use NumPy for whole-array math and a small thread-grid simulator that runs a kernel function once for every simulated thread, so you can see index arithmetic, memory access patterns and counts really executing.
  • Stand-ins model ideas, not speed. A simulated kernel is slower than a real one by orders of magnitude, and any "cost" numbers come from simple models you can read and change. They never claim to be benchmarks.

Here is a preview of the real thing, so you recognize it later. Do not worry about the details yet: Your First Kernel explains every symbol.

C++
#include <cstdio>

__global__ void hello() {
    printf("Hello from thread %d of block %d\n", threadIdx.x, blockIdx.x);
}

int main() {
    hello<<<2, 3>>>();          // 2 blocks of 3 threads each
    cudaDeviceSynchronize();    // wait for the GPU to finish
    return 0;
}

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main design goal of a GPU compared with a CPU?

    A GPU trades single-thread speed for a very large number of simple lanes, so it completes huge amounts of parallel work per second.
  2. 2. Which task is the best fit for a GPU?

    Pixel filters are data parallel: each output depends only on its own input, so thousands of threads can work at once.
  3. 3. Why does a GPU program launch far more threads than the chip has arithmetic lanes?

    Without large caches, the GPU hides waiting time by always having other threads ready to run.
  4. 4. How are the runnable examples in this tutorial related to real CUDA?

    The site has no GPU or nvcc, so Python simulators and NumPy show how indexing and memory ideas work without claiming real speed.

Summary

  • CPUs minimize the latency of one task; GPUs maximize throughput across thousands of tasks.
  • GPUs hide memory waiting time with massive thread counts rather than large caches.
  • Data parallelism (the same operation on many independent items) is the sweet spot.
  • Small, sequential, branchy or irregular problems usually belong on the CPU, and a tuned library often beats a hand-written kernel.
  • CUDA, introduced by NVIDIA in 2006-2007, lets you program the GPU in C++; here the real code is read-only and the runnable parts are honest Python stand-ins.

Next: install the tools and learn to read device information in Setting Up CUDA.