Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 45 ยท Tensors in Depth

Indexing, Slicing and Reshaping

Index and slice PyTorch tensors, use boolean masks, and reshape with view, reshape, permute, squeeze, unsqueeze, cat and stack while understanding shared storage.

Deep learning code spends a surprising amount of effort moving numbers between shapes: a batch of images becomes a flat matrix, channels move to a different axis, two batches join into one. PyTorch gives you a small, consistent toolbox for this. The one concept that explains most surprises is shared storage: many shape operations return a new tensor that looks at the same memory.

This lesson covers indexing and slicing, boolean masks, view and reshape, permute, squeeze and unsqueeze, joining tensors, and contiguity. It mirrors Indexing and Slicing, Reshaping Arrays and Views and Copies in the NumPy track.

Indexing and slicing

Indexing works like NumPy. Each comma-separated entry addresses one dimension.

Python
import torch

x = torch.arange(12).reshape(3, 4)
# tensor([[ 0,  1,  2,  3],
#         [ 4,  5,  6,  7],
#         [ 8,  9, 10, 11]])

x[0]          # first row: tensor([0, 1, 2, 3])
x[1, 2]       # tensor(6), a 0-D tensor
x[1, 2].item()  # 6, a Python number
x[:, 1]       # second column: tensor([1, 5, 9])
x[1:, :2]     # rows 1.., columns 0..1
x[..., -1]    # last column, any number of leading dimensions

A basic slice returns a view: writing to it changes x. Use .item() to get a plain Python number out of a one-element tensor (and .tolist() for a list).

Boolean masks and index tensors

A comparison produces a boolean tensor you can use as a filter or a switch:

Python
mask = x > 6
x[mask]                     # 1-D tensor of the selected values (always a copy)
x[x % 2 == 0] = -1          # assign through a mask, in place
torch.where(x > 6, x, 0)    # choose per element: x where true, else 0

idx = torch.tensor([2, 0])
x[idx]                      # rows 2 and 0, in that order (a copy)

Masks and integer index tensors are called advanced indexing, and they always return a copy, unlike basic slices.

view, reshape and contiguity

A tensor is a small header (shape, strides, dtype) pointing at a flat block of memory called its storage. view gives the same storage a new shape without moving anything:

Python
a = torch.arange(6)
b = a.view(2, 3)
b[0, 0] = 99
print(a)      # tensor([99,  1,  2,  3,  4,  5])  shared!

view only works when the memory layout allows it. A freshly made tensor is contiguous: its elements sit in row-major order. After transpose or permute the strides no longer match, so view fails with an error that tells you to use reshape or call .contiguous():

Python
m = torch.arange(6).reshape(2, 3)
t = m.t()                 # transpose: a view with swapped strides
t.is_contiguous()         # False
# t.view(6)               # RuntimeError: view size is not compatible...
t.reshape(6)              # works: copies when it has to
t.contiguous().view(6)    # the same, spelled out

The rule of thumb: reshape is safe, view is strict. reshape returns a view when possible and silently copies otherwise, so do not rely on it sharing memory. One dimension may be -1, meaning "work it out":

Python
imgs = torch.randn(32, 3, 28, 28)
flat = imgs.reshape(32, -1)      # shape (32, 2352)

permute, transpose, squeeze and unsqueeze

permute reorders all dimensions at once; transpose(a, b) swaps two. Both are views:

Python
imgs = torch.randn(32, 3, 28, 28)       # batch, channels, height, width
hwc = imgs.permute(0, 2, 3, 1)           # batch, height, width, channels

unsqueeze(dim) inserts a size-1 dimension and squeeze() removes size-1 dimensions. They are how you make shapes line up for broadcasting:

Python
v = torch.tensor([1.0, 2.0, 3.0])   # shape (3,)
v.unsqueeze(0).shape                 # (1, 3), a row
v.unsqueeze(1).shape                 # (3, 1), a column
v[:, None].shape                     # (3, 1), same as unsqueeze(1)
torch.zeros(1, 3, 1).squeeze().shape # (3,)

View or copy?

View Same memory. Slices, view, permute, t.
Copy New memory. Masks, index tensors, clone.

Writing through a view changes the original.

1 / 2

Joining tensors: cat and stack

torch.cat joins along an existing dimension; torch.stack creates a new one. The inputs must otherwise have identical shapes.

Python
a = torch.zeros(2, 3)
b = torch.ones(2, 3)

torch.cat([a, b], dim=0).shape     # (4, 3)
torch.cat([a, b], dim=1).shape     # (2, 6)
torch.stack([a, b]).shape          # (2, 2, 3), a new leading dimension
torch.stack([a, b], dim=1).shape   # (2, 2, 3), new dimension in position 1

Stacking is how a list of single images becomes a batch. Splitting goes the other way with chunk, split and unbind.

A live stand-in: views and copies in NumPy

NumPy arrays follow the same view/copy rules, so you can watch shared storage work. The tool is np.shares_memory.

Python Views share memory, copies do not

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last two lines show the idea behind PyTorch's reshape: a transposed array cannot be flattened without moving data, so a copy is made.

A live stand-in: reshaping, stacking and axes

Python reshape, permute, stack, cat

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Everything above is NumPy, but only the function names differ in PyTorch: transpose becomes permute, concatenate becomes cat, and axis becomes dim.

Warning: Do not assume reshape shares memory. If you need a guaranteed copy, call .clone(). If you need guaranteed sharing, use view and let it raise an error when it cannot.

Rank, dimension and axis: three words, careful use

These words are used loosely, so pin them down:

  • The rank of a tensor (in the machine learning sense) is its number of dimensions: a scalar has rank 0, a vector rank 1, a matrix rank 2, a batch of RGB images (N, C, H, W) rank 4. PyTorch spells it x.ndim or x.dim(). (Do not confuse it with the matrix rank of linear algebra, the number of independent rows.)
  • A dimension (or axis) is one of those directions, identified by an index: dim=0 is the first, dim=-1 the last. "Dimension" is also used for the size along an axis, so be explicit: "dimension 1 has size 3".
  • The shape is the tuple of sizes, x.shape, and x.numel() is their product.

For x = torch.zeros(2, 3, 4) you have x.ndim == 3, x.shape == (2, 3, 4), x.size(1) == 3 and x.numel() == 24. In a reduction like x.sum(dim=1) the dimension you name is the one that disappears (or collapses to size 1 with keepdim=True).

Sparse tensors

A sparse tensor stores only the nonzero entries and their positions, which saves memory and time when most values are zero (adjacency matrices of graphs, one-hot or bag-of-words features, pruned weights). The COO (coordinate) format keeps two things: an indices tensor of shape (ndim, nnz) with the position of every stored value, and a values tensor of length nnz. Everything not listed is zero.

Python
import torch

indices = torch.tensor([[0, 1, 2],     # row of each stored value
                        [2, 0, 1]])    # column of each stored value
values = torch.tensor([3.0, 4.0, 5.0])
s = torch.sparse_coo_tensor(indices, values, size=(3, 3))

print(s.to_dense())                    # the full 3x3 matrix with zeros filled in
print(torch.sparse.mm(s, torch.ones(3, 2)))     # sparse @ dense matrix product
dense = torch.tensor([[0., 0., 3.], [4., 0., 0.], [0., 5., 0.]])
s2 = dense.to_sparse()                 # convert the other way

Other layouts exist (to_sparse_csr() for fast row access and matmul). Operations that involve sparse tensors are a subset of the dense ones: sparse with dense multiplication, addition of two sparse tensors, transposes and sums work, while many indexing and elementwise operations do not. Sparse only pays off when the data is very sparse; a half-empty matrix is faster dense. Also note that nn.Embedding(..., sparse=True) produces sparse gradients that only some optimizers (SGD, SparseAdam, Adagrad) support.

Nested (ragged) tensors

Real data is often ragged: sentences of different lengths, point clouds with different counts. The usual fix is to pad to a common length and carry a mask (see the padding discussion in Recurrent Networks). A nested tensor instead groups tensors of different lengths into one object that stores them back to back with offsets, so no compute is wasted on padding:

Python
import torch

nt = torch.nested.nested_tensor([torch.tensor([1., 2., 3.]),
                                 torch.tensor([4., 5.]),
                                 torch.tensor([6.])])
print(nt.is_nested)            # True
print(nt.unbind())             # back to a tuple of ordinary tensors
padded = torch.nested.to_padded_tensor(nt, 0.0)     # shape (3, 3) with zero padding

The nested tensor API is a prototype with a smaller set of supported operations than dense tensors, so check the current docs before relying on a specific operation. When in doubt, padded tensors plus a mask are the portable solution. The next editor builds both structures with NumPy: the sparse COO format and the "flat values plus row offsets" layout that ragged tensors use.

Python COO sparse storage and ragged rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these returns a copy rather than a view?

    Boolean mask indexing is advanced indexing and always copies. Basic slices, permute and view share storage.
  2. 2. Why can .view() fail after .t() (transpose)?

    Transposing swaps strides without moving data, so the layout no longer matches row-major order. Use reshape or contiguous().
  3. 3. What does torch.stack([a, b]) do when a and b both have shape (2, 3)?

    stack adds a new dimension, so the result is (2, 2, 3). cat would join along an existing one.
  4. 4. What does v.unsqueeze(1) do to a tensor of shape (5,)?

    unsqueeze(dim) inserts a size-1 dimension at position dim, so unsqueeze(1) produces a column of shape (5, 1).

Summary

  • Indexing and slicing follow NumPy; .item() pulls out a Python number.
  • Basic slices, view, permute, transpose and unsqueeze return views on the same storage.
  • Boolean masks and index tensors return copies.
  • view needs contiguous memory; reshape copies when it must; .contiguous() makes a tidy copy.
  • cat joins along an existing dimension and stack creates a new one.
  • Use .clone() when you need an independent copy.

Next: combine tensors with arithmetic in Tensor Math and Broadcasting.