Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 45 ยท Tensors in Depth

Dtypes, Precision and NumPy Interop

Learn PyTorch dtypes (float32, float64, float16, bfloat16, ints), type promotion, to(), and how from_numpy and numpy() share memory, with a live precision demo.

Every tensor stores one kind of number, its dtype. The dtype decides how much memory the tensor uses, how precisely it represents values, and which operations accept it. Training neural networks is a constant negotiation between precision (more bits, more accurate) and cost (fewer bits, faster and smaller).

This lesson tours the dtypes you will actually use, explains how PyTorch picks a result type when you mix them, shows to() for conversions, and covers the memory-sharing behavior when moving between PyTorch and NumPy. The ideas are the same as in NumPy's Data Types lesson.

The dtypes that matter

dtype Bits Typical use
torch.float32 32 The default for model weights and activations
torch.float64 64 Scientific code, gradient checks
torch.float16 16 Mixed-precision training and inference on GPUs
torch.bfloat16 16 Mixed precision with float32's range
torch.int64 64 Class labels, indices (the default integer)
torch.int32, int8, uint8 32, 8, 8 Compact storage, quantization, raw images
torch.bool 8 Masks from comparisons
Python
import torch

x = torch.tensor([1.5, 2.5])
print(x.dtype)            # torch.float32
print(torch.tensor([1, 2]).dtype)       # torch.int64
print(torch.tensor([True, False]).dtype)  # torch.bool
print(torch.tensor([1, 2.5]).dtype)     # torch.float32: one float promotes the list

Two 16-bit float formats exist because they split their bits differently. float16 has more precision but a tiny range (its largest value is about 65,504), so large numbers overflow to infinity. bfloat16 keeps the same range as float32 and gives up precision, which is why it is popular for training: it rarely overflows.

Fewer bits, trade-offs

float64 Most precise. Slowest, biggest.
float32 The everyday default.
float16 Fast, small, can overflow.
bfloat16 Float32 range, less precision.
1 / 2

Converting with to()

to() converts dtype, device, or both. It returns the same tensor when nothing changes, otherwise a copy:

Python
x = torch.arange(4)                 # int64
f = x.to(torch.float32)             # or x.float()
d = x.to(dtype=torch.float64)       # or x.double()
h = f.to(torch.float16)             # or f.half()
b = f.to(torch.bfloat16)
i = torch.tensor([1.9, -1.9]).to(torch.int64)   # tensor([ 1, -1]): truncates toward zero

y = torch.zeros(2, dtype=torch.float64)   # choose dtype at creation
z = torch.zeros_like(x, dtype=torch.float32)

Converting a float to an int truncates, it does not round. Use torch.round first if you want rounding.

Type promotion

When you combine tensors of different dtypes, PyTorch promotes to a type that can hold both:

Python
ints = torch.tensor([1, 2, 3])
floats = torch.tensor([0.5, 0.5, 0.5])

(ints + floats).dtype      # torch.float32
(ints + 1).dtype           # torch.int64: a Python int keeps the tensor's dtype
(ints + 1.5).dtype         # torch.float32: a Python float promotes an int tensor
(ints / 2).dtype           # torch.float32: true division of ints gives floats
(torch.tensor([1.0], dtype=torch.float16) + torch.tensor([1.0])).dtype   # float32

In short: a float beats an int, a wider type beats a narrower one of the same kind, and a plain Python scalar does not widen a tensor of the same category. Some operations refuse to mix: for example, a loss like CrossEntropyLoss wants int64 class labels and float logits, and a layer fed integers raises a dtype error.

Sharing memory with NumPy

torch.from_numpy and tensor.numpy() do not copy. They give you two views of the same memory:

Python
import numpy as np

arr = np.array([1.0, 2.0, 3.0])
t = torch.from_numpy(arr)
arr[0] = 100.0
print(t)                  # tensor([100., 2., 3.], dtype=torch.float64)

t2 = torch.ones(3)
n = t2.numpy()
t2.add_(1)
print(n)                  # [2. 2. 2.]

Notice two things. The tensor from NumPy is float64, because NumPy's default is float64; convert with .float() if your model expects float32 (that conversion makes a copy). And .numpy() only works for CPU tensors that do not require gradients. For a tensor in a graph, use x.detach().cpu().numpy(). Use torch.tensor(arr) or torch.as_tensor(arr) depending on whether you want an independent copy or sharing where possible.

Warning: Shared memory is fast but surprising. If you load data with NumPy and keep modifying the array while a tensor still uses it, your training data silently changes. Call .clone() or torch.tensor(arr) when you want independence.

Live demo: how precision behaves

You cannot run PyTorch dtypes here, but NumPy has float16, float32 and float64, and they behave like the PyTorch types (NumPy has no bfloat16). This demo shows what limited precision does.

Python Float precision: 16, 32 and 64 bits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see that float16 cannot store 2048 + 1 (the spacing between neighbouring values at that size is 2), that 70000 becomes inf, and that 0.1 gets more digits of error as the type narrows. This is why mixed-precision training keeps a float32 copy of the weights and accumulates sums in higher precision, as you will see in GPUs, Devices and Mixed Precision.

Live demo: type promotion and sharing

NumPy promotes types with almost the same rules, and its array/array views behave like the PyTorch-NumPy bridge.

Python Promotion and shared memory in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the one place where NumPy and PyTorch differ: PyTorch promotes int64 + float32 to float32, while NumPy widens it to float64. Always check .dtype instead of assuming.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the dtype of torch.tensor([1, 2, 3])?

    Python integers become int64 by default. Floats become float32.
  2. 2. Why is bfloat16 popular for training?

    bfloat16 keeps float32's exponent size, trading away precision, so large values do not overflow like they can in float16.
  3. 3. What happens to arr when you run t = torch.from_numpy(arr) and then t[0] = 5?

    from_numpy shares memory with the array, so a write through one is visible through the other.
  4. 4. What does torch.tensor([1.9, -1.9]).to(torch.int64) give?

    Float to int conversion truncates toward zero; it does not round.

Summary

  • A dtype sets a tensor's precision, memory use and which operations accept it.
  • Defaults: float32 for floats, int64 for integers, bool for comparisons.
  • float16 is compact but has a small range; bfloat16 keeps float32's range with less precision.
  • to(), .float(), .double() and .half() convert dtypes; float to int truncates.
  • Mixed dtypes are promoted: floats beat ints, and Python scalars do not widen tensors of the same kind.
  • from_numpy and .numpy() share memory; use .clone() for independence and detach().cpu() for tensors in a graph.

Next: see where gradients come from in Computational Graphs and the Chain Rule.