Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 31 ยท Memory

Moving Data: Pinned, Managed and Unified Memory

Why PCIe copies dominate many CUDA programs, and how pinned memory, cudaMallocManaged, prefetching and zero-copy change the cost, with a runnable cost model.

So far we have talked about memory on the GPU. But your data starts in the host's RAM, and the host and the device are connected by a link that is far slower than the GPU's own memory. A GPU can read its memory at several hundred GB/s or more. A PCIe 4.0 x16 link moves roughly 25 GB/s in practice. Many first CUDA programs discover this the hard way: the kernel takes a fraction of a millisecond, but copying the data in and out takes tens of milliseconds, so the program is slower than a plain CPU loop.

This lesson explains the costs of moving data and the tools CUDA gives you to control them: pinned memory for faster, asynchronous copies, managed (unified) memory for automatic migration, prefetching, and zero-copy access. It ends with the most important rule: move less data, less often.

Everything below is read-only CUDA plus a Python cost model. The site cannot copy to a GPU. The model's numbers (link speed, per-call overhead, page fault cost) are illustrative round values chosen to show the shape of the trade-offs; real values depend on the card, the bus, the driver and the operating system. Always measure your own setup.

The shape of the problem

flowchart LR
    H["Host RAM<br/>CPU side"] -- "PCIe or NVLink<br/>about 25 GB/s typical PCIe 4.0" --> D["Device memory<br/>GPU side<br/>hundreds of GB/s to TB/s"]
    D --> C["GPU cores"]

The link between host and device is much narrower than the GPU's own memory bus.

A copy's time is roughly fixed overhead + bytes / bandwidth. The fixed part is a few microseconds per call, so many tiny copies are wasteful, and the bandwidth part dominates for big transfers. The model below puts numbers on a vector addition of two 256 MiB arrays: copy both to the device, run the kernel, copy the result back.

Python Transfer time versus kernel time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this model the kernel is only 3 percent of the job; the other 97 percent is moving data. Vector addition does one arithmetic operation per three memory accesses, so it is a poor candidate for the GPU unless the data is already there. A GPU pays off when the kernel does a lot of work per byte moved (matrix multiplication, simulation steps, neural network layers) or when the data stays on the device across many kernels. The last two lines show the fixed-cost effect: a thousand 4 KiB copies cost about 55 times as much as one copy of the same bytes, because of the per-call overhead.

Pinned (page-locked) memory

Memory you get from malloc or new is pageable: the operating system may move it around or swap it out. The GPU's copy engine (DMA) can only read memory that stays at a fixed physical address, so for a pageable buffer the driver first copies your data into a hidden pinned staging buffer and then DMAs it. That extra copy is why pageable transfers are slower.

You can allocate pinned memory yourself:

C++
float* h_data;
cudaMallocHost(&h_data, bytes);        // page-locked host allocation
// ... fill h_data ...
cudaMemcpy(d_data, h_data, bytes, cudaMemcpyHostToDevice);   // faster than pageable
// ...
cudaFreeHost(h_data);

Pinned memory has two benefits: higher bandwidth, and it is the requirement for truly asynchronous copies with cudaMemcpyAsync, which let a copy run while the GPU computes (Streams and Events). The cost is that pinned pages cannot be swapped, so allocating too much starves the operating system and other programs. Pin the buffers you use for repeated transfers, not everything.

Managed (unified) memory

Writing cudaMalloc, cudaMemcpy and cudaFree for every array is tedious. cudaMallocManaged allocates one pointer that is valid on both the host and the device:

C++
float* data;
cudaMallocManaged(&data, bytes);       // one pointer, usable by CPU and GPU

for (int i = 0; i < n; ++i) data[i] = 1.0f;      // CPU writes it

scale<<<blocks, threads>>>(data, n);   // GPU reads and writes the same pointer
cudaDeviceSynchronize();               // wait before the CPU touches it again

printf("%f\n", data[0]);               // CPU reads the result
cudaFree(data);

There is no explicit copy in the source, but data still has to travel. On modern GPUs on Linux the driver migrates memory in pages: when the GPU touches a page that currently lives in host RAM, it triggers a page fault, the driver moves the page across the link, and execution continues. When the CPU touches it later, the page migrates back. Managed memory is excellent for getting a program working, for data larger than device memory (the driver can oversubscribe), and for irregular access patterns where you cannot predict what is needed. The costs are the fault handling and the lack of control.

Two details are important:

  • After a kernel launch the call returns immediately, so the CPU must call cudaDeviceSynchronize() (or use a stream sync) before it touches the managed data, otherwise the access races with the kernel.
  • Behaviour varies by platform. Older GPUs and some Windows setups migrate the whole allocation at launch instead of faulting page by page.

Prefetch and advise

If you know where data will be used, tell the driver:

C++
int device = 0;
cudaGetDevice(&device);
cudaMemPrefetchAsync(data, bytes, device);          // move to the GPU before the kernel
scale<<<blocks, threads>>>(data, n);
cudaMemPrefetchAsync(data, bytes, cudaCpuDeviceId); // move back for the CPU

cudaMemAdvise adds hints such as "this range is mostly read" (so copies can be duplicated) or "prefer this location". Newer CUDA versions use a descriptor-style argument for the target location in some prefetch calls, so check the header of the toolkit you have installed.

The model below compares moving 256 MiB by page faults, by prefetch and by an explicit copy.

Python Page faults versus prefetch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With these assumptions the faulting version is more than ten times slower than prefetch, and prefetching essentially matches an explicit copy: the bytes still have to cross the link, but the per-page overhead disappears. The ratio is model-dependent, yet the lesson holds on real systems: unified memory is a convenience, and prefetching restores most of the performance.

Zero-copy access

A third option is to let the GPU read host memory directly while the kernel runs. You allocate mapped pinned memory and get a device pointer to it:

C++
float* h;
cudaHostAlloc(&h, bytes, cudaHostAllocMapped);
float* d;
cudaHostGetDevicePointer(&d, h, 0);
kernel<<<blocks, threads>>>(d, n);     // each access travels over PCIe

Nothing is copied up front, but every access goes over the link. That is perfect when each element is read exactly once, such as streaming a large input that you never reuse, or tiny data like a flag. It is terrible when the kernel touches the data repeatedly. The next model counts how many times the kernel reads a 256 MiB array.

Python Zero-copy versus copy-then-use

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For one pass the two strategies tie (copying costs about the same as streaming it once). After that, the zero-copy kernel pays the link price on every pass, and copying once then reading from fast device memory wins by a growing margin. Integrated GPUs that share RAM with the CPU are the exception: there zero-copy is natural and cheap because there is no PCIe link.

Strategy: move less, overlap the rest

  1. Keep data on the device. Chain kernels without bringing intermediate results back to the host.
  2. Batch. One large copy beats many small ones.
  3. Pin repeated buffers, and use cudaMemcpyAsync in streams to overlap copies with computation.
  4. Choose the programming model deliberately: explicit copies for performance-critical code, managed memory with prefetch when convenience matters, zero-copy for read-once data.
  5. Compute more per byte so the transfer cost is amortized.

Overlap is the final trick. If you split the work into chunks, the copy of chunk i + 1 can run while chunk i computes. The next model shows the effect with a 4 ms copy and a 1.5 ms kernel for the whole job.

Python Pipelining copy and compute

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pipelined time approaches the cost of the slower activity (the 4 ms copy) as chunks increase. It can never beat the copy time alone, which is why reducing the data moved is still the first thing to try. You will build this with real streams in the Concurrency chapter.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is copying from pinned host memory faster than from pageable memory?

    Pageable memory may move, so the driver stages it through a pinned buffer first. Page-locked memory can be read by the copy engine directly.
  2. 2. After launching a kernel that uses cudaMallocManaged memory, what must the CPU do before reading the data?

    Kernel launches are asynchronous. Without a synchronization, the CPU would race with the kernel.
  3. 3. When is zero-copy host memory a reasonable choice?

    Every zero-copy access crosses the PCIe link. That is fine for read-once data, but costly for repeated access.
  4. 4. What does cudaMemPrefetchAsync mainly improve for managed memory?

    Prefetching migrates the range in bulk ahead of time, so the kernel does not stall on many individual page faults.
  5. 5. Which change usually helps the most when a program is dominated by transfer time?

    If transfers dominate, kernel tuning barely matters. Reducing the bytes moved, or overlapping the transfers, is the lever.

Summary

  • Host-device transfers are slow compared with device memory, and each copy call has a fixed overhead; batch copies and move less.
  • Pinned memory (cudaMallocHost) gives faster copies and enables asynchronous transfers, but do not pin excessively.
  • cudaMallocManaged gives one pointer for CPU and GPU; pages migrate on demand, so synchronize before the CPU touches the data and consider prefetching.
  • Zero-copy suits read-once data and integrated GPUs; it is slow for repeated access.
  • Overlap copies and kernels with streams, but the best optimization is to reduce the data moved.

Now that you know where data lives and how it moves, the next chapter deals with many threads working on shared data safely: Synchronization and Atomics.