Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 45 ยท Tensors in Depth

Tensor Math and Broadcasting

Do tensor math in PyTorch: elementwise ops, matmul, reductions with dim and keepdim, broadcasting rules, in-place operations and einsum, with live NumPy demos.

Once you can create and reshape tensors, the next skill is computing with them. Almost everything a neural network does is a handful of operations: elementwise arithmetic, matrix multiplication, reductions such as sums and means, and broadcasting, the rule that lets tensors of different shapes combine.

This lesson shows each in PyTorch (read-only) and then runs the same ideas in NumPy, which uses identical rules. If you have read Broadcasting in the NumPy track, the rules will be familiar.

Elementwise operations

Operators work element by element on tensors of the same shape (or shapes that broadcast):

Python
import torch

a = torch.tensor([1.0, 2.0, 3.0])
b = torch.tensor([10.0, 20.0, 30.0])

a + b            # tensor([11., 22., 33.])
a * b            # tensor([10., 40., 90.])  (not matrix multiplication)
a ** 2           # tensor([1., 4., 9.])
torch.exp(a)     # also a.exp()
torch.sqrt(a)
torch.clamp(a, min=1.5)    # tensor([1.5000, 2.0000, 3.0000])

* is always elementwise. Matrix multiplication has its own operator.

Matrix multiplication

Python
A = torch.randn(2, 3)
B = torch.randn(3, 4)

C = A @ B                 # shape (2, 4); same as torch.matmul(A, B)
v = torch.randn(3)
A @ v                     # matrix times vector: shape (2,)
(torch.randn(8, 2, 3) @ B).shape   # batched: (8, 2, 4)

The inner dimensions must match: (2, 3) @ (3, 4) works because both are 3. With more than two dimensions, matmul treats the leading ones as batch dimensions and multiplies matrix by matrix. A linear layer in a network is exactly x @ W.T + b.

Reductions: sum, mean, max along a dimension

A reduction collapses one or more dimensions. The dim argument says which one disappears:

Python
x = torch.tensor([[1.0, 2.0, 3.0],
                  [4.0, 5.0, 6.0]])

x.sum()                        # tensor(21.)
x.sum(dim=0)                   # tensor([5., 7., 9.])    collapse rows
x.sum(dim=1)                   # tensor([ 6., 15.])      collapse columns
x.mean(dim=1, keepdim=True)    # shape (2, 1): tensor([[2.], [5.]])
x.max(dim=1)                   # (values=tensor([3., 6.]), indices=tensor([2, 2]))
x.argmax(dim=1)                # tensor([2, 2])

Think of dim=1 as "walk along dimension 1 and combine". keepdim=True keeps the collapsed dimension with size 1, so the result broadcasts cleanly against the original. NumPy spells the keyword keepdims and axis instead of dim (PyTorch also accepts axis as an alias in many functions).

Broadcasting rules

When two tensors have different shapes, PyTorch compares them from the last dimension backwards:

  1. Missing leading dimensions are treated as size 1.
  2. Two sizes are compatible if they are equal or one is 1.
  3. A size-1 dimension is stretched to match; no data is copied.
  4. Anything else is an error.

The broadcasting recipe

Align right Compare shapes from the last dimension.
Pad with 1s Missing leading dimensions count as 1.
Stretch 1s Size-1 dimensions repeat to match.
Else error Different and not 1 cannot combine.
1 / 2
Python
x = torch.randn(32, 10)          # a batch of 32 samples, 10 features
mean = x.mean(dim=0)             # (10,)
x_centered = x - mean            # (32, 10) - (10,) broadcasts

row_sums = x.sum(dim=1, keepdim=True)   # (32, 1)
x_normalized = x / row_sums             # (32, 10) / (32, 1)

If you wrote x.sum(dim=1) without keepdim=True, the shape (32,) would fail to divide against (32, 10): aligned on the right it compares 32 with 10. This is the most common broadcasting error in practice. Silent mistakes are possible too: (5,) combined with (5, 1) quietly gives a (5, 5) grid.

In-place operations

A method ending in an underscore changes the tensor instead of returning a new one:

Python
x = torch.ones(3)
x.add_(5)       # x is now tensor([6., 6., 6.])
x.mul_(2)
x.zero_()

In-place operations save memory, and optimizers use them to update weights. They can break gradient computation if they overwrite a value autograd still needs, so avoid them on tensors that take part in a graph unless you know why. The autograd lesson explains the error you would see.

einsum: one notation for many products

torch.einsum writes a tensor operation as a string naming each axis with a letter. Repeated letters are multiplied together, and letters missing from the output are summed:

Python
torch.einsum("ij,jk->ik", A, B)       # matrix multiplication
torch.einsum("ij->ji", A)             # transpose
torch.einsum("ij->i", A)              # sum over columns
torch.einsum("bij,bjk->bik", X, Y)    # batched matrix multiplication
torch.einsum("i,i->", u, v)           # dot product

It reads slowly at first, then becomes the clearest way to write attention and other multi-axis products. The stand-ins below use np.einsum, which has the same syntax.

Live demos in NumPy

Python Broadcasting: centering and normalizing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The keepdims=True line is exactly PyTorch's keepdim=True. Without it, the (4,) shape cannot be aligned on the right with (4, 3).

Python matmul, reductions and einsum agree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both examples are real NumPy running live. Swap np. for torch. and axis for dim and the PyTorch versions behave the same way.

Tip: When a broadcast surprises you, write the two shapes one above the other, right-aligned, and check each column. It takes ten seconds and finds the bug.

Bitwise and logical operations

Integer and boolean tensors support bitwise operations that work on the binary representation of each element. They are handy for masks, flags, hashing tricks and packing several boolean values into one integer.

Operation Operator Function Meaning on each pair of bits
AND a & b torch.bitwise_and(a, b) 1 only if both are 1
OR a | b torch.bitwise_or(a, b) 1 if either is 1
XOR a ^ b torch.bitwise_xor(a, b) 1 if exactly one is 1
NOT (invert) ~a torch.bitwise_not(a) flips every bit
Left shift a << n torch.bitwise_left_shift(a, n) multiplies by 2**n
Right shift a >> n torch.bitwise_right_shift(a, n) floor-divides by 2**n
Python
import torch

a = torch.tensor([12, 10, 5], dtype=torch.int32)   # 1100, 1010, 0101
b = torch.tensor([10, 6, 3], dtype=torch.int32)    # 1010, 0110, 0011
print(a & b)     # tensor([ 8,  2,  1], dtype=torch.int32)
print(a | b)     # tensor([14, 14,  7], dtype=torch.int32)
print(a ^ b)     # tensor([ 6, 12,  6], dtype=torch.int32)
print(~a)        # tensor([-13, -11,  -6], dtype=torch.int32)  (two's complement)
print(a << 2)    # tensor([48, 40, 20], dtype=torch.int32)

These only work on integer and bool dtypes; floats raise an error. For boolean tensors, &, |, ^ and ~ act as logical AND, OR, XOR and NOT, which is exactly how you combine masks: mask = (x > 0) & (x < 10). Use parentheses, because & binds more tightly than comparisons. torch.logical_and, logical_or, logical_xor and logical_not also accept non-boolean inputs and treat nonzero as true. A common use is counting the bits that differ between two codes, the Hamming distance: XOR them, then count the ones.

Python Bitwise operations and bit counting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does x.sum(dim=0) do to a tensor of shape (2, 3)?

    dim=0 is the dimension that disappears, so the two rows are added together and a vector of length 3 remains.
  2. 2. Which shape pair fails to broadcast?

    Aligned on the right, 5 is compared with 8, which are different and neither is 1.
  3. 3. What is keepdim=True for?

    A reduction normally removes the dimension. keepdim=True leaves it at size 1, giving shapes like (32, 1).
  4. 4. What does a trailing underscore in x.add_(1) mean?

    Trailing-underscore methods mutate the tensor and return it. That saves memory but can interfere with autograd.
  5. 5. What does torch.einsum("ij,jk->ik", A, B) compute?

    The shared letter j is multiplied and summed out, leaving i and k: the matrix product.

Summary

  • Operators like + and * are elementwise; @ (or matmul) is matrix multiplication.
  • Reductions take dim for the axis to collapse and keepdim=True to keep it as size 1.
  • Broadcasting aligns shapes from the right, pads with 1s and stretches size-1 dimensions.
  • Methods ending in _ work in place; be careful with them near autograd.
  • einsum writes products and sums as a small string of axis letters.
  • NumPy follows the same rules, so its examples carry over directly.

Next: control number precision and convert between libraries in Dtypes, Precision and NumPy Interop.