Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 40 ยท Tensors and Automatic Differentiation

Tensor Operations and Broadcasting

Master tensor math for deep learning: element-wise operations, matmul, reductions with axis, softmax, argmax and broadcasting rules, with NumPy stand-ins.

Neural networks are built from a small set of tensor operations: element-wise arithmetic, matrix multiplication, reductions (sums, means, maxima) and broadcasting, which lets tensors of different shapes combine. If you understand these, you understand what every layer does. This lesson shows the TensorFlow spellings in read-only code and lets you run the equivalent NumPy to check results and shapes.

Element-wise operations

Arithmetic operators work on corresponding elements of tensors with the same shape (or shapes that broadcast, see below).

Python
a = tf.constant([[1.0, 2.0], [3.0, 4.0]])
b = tf.constant([[10.0, 20.0], [30.0, 40.0]])

a + b          # tf.add
a * b          # tf.multiply (element-wise, NOT matrix product)
a ** 2         # tf.pow
tf.sqrt(a)
tf.exp(a)
tf.maximum(a, 2.5)

The classic mistake: * multiplies element by element. Matrix multiplication needs tf.matmul or the @ operator.

Python Stand-in: element-wise math (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Matrix multiplication

For matrices A of shape (m, n) and B of shape (n, p), the product A @ B has shape (m, p). The inner sizes must match. Entry (i, j) is the dot product of row i of A with column j of B. A Dense layer is exactly this: (batch, n_in) @ (n_in, n_out) -> (batch, n_out).

Python
x = tf.ones((32, 784))
w = tf.ones((784, 10))
y = tf.matmul(x, w)         # or x @ w
print(y.shape)              # (32, 10)

Batched matmul also works: a rank-3 tensor (batch, m, n) times (batch, n, p) multiplies each matrix pair, which attention layers use.

flowchart LR
    X["x<br/>(32, 784)"] --> M{"matmul"}
    W["w<br/>(784, 10)"] --> M
    M --> Y["y<br/>(32, 10)"]

The inner dimension (784) disappears; the outer ones remain.

Python Stand-in: matmul shapes and a dot-product check (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reductions and the axis argument

A reduction collapses one or more axes into a single number: sum, mean, max, min, product. The axis argument names the axis that disappears.

Python
t = tf.constant([[1.0, 2.0, 3.0],
                 [4.0, 5.0, 6.0]])      # shape (2, 3)

tf.reduce_sum(t)                  # 21.0 (all axes)
tf.reduce_sum(t, axis=0)          # [5. 7. 9.]  shape (3,): collapse rows
tf.reduce_sum(t, axis=1)          # [ 6. 15.]   shape (2,): collapse columns
tf.reduce_mean(t, axis=1, keepdims=True)  # shape (2, 1)
tf.argmax(t, axis=1)              # index of the max per row: [2 2]

keepdims=True keeps the collapsed axis with size 1, which makes later broadcasting work. A loss is usually a mean over the batch axis, and a classifier's predicted class is an argmax over the class axis.

Python Stand-in: reductions along axes (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Broadcasting

When shapes differ, TensorFlow (like NumPy) compares them from the last axis backwards. Two sizes are compatible if they are equal or one of them is 1; the size-1 axis is stretched. Missing leading axes count as 1.

Shapes Result Why
(3, 4) + (4,) (3, 4) the vector is added to every row
(3, 4) + (3, 1) (3, 4) the column is added to every column
(3, 1) + (1, 4) (3, 4) outer combination
(3, 4) + (3,) error 4 vs 3 is incompatible

The bias in a layer is broadcasting at work: x @ w has shape (batch, units) and b has shape (units,), so the same bias is added to every example.

Python Stand-in: broadcasting and a shape error (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Softmax and friends

A few composite operations appear constantly. Softmax turns a vector of raw scores (logits) into probabilities that are positive and sum to 1: exp(z) / sum(exp(z)). Subtracting the maximum first keeps it numerically stable.

Python
logits = tf.constant([[2.0, 1.0, 0.1]])
tf.nn.softmax(logits)          # about [[0.659 0.242 0.099]]
Python Stand-in: a stable softmax (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Operations on a GPU

If a GPU is present, TensorFlow places large operations there automatically. You rarely manage devices by hand. What matters is that you keep work in tensor operations: a Python for loop over elements is thousands of times slower than one tf.matmul. Always prefer a vectorized operation.

Common mistakes

  • Using * when you meant @.
  • Mixing dtypes, for example adding an int32 tensor to a float32 tensor (an error in TensorFlow, not a silent upcast).
  • Forgetting keepdims, so a (batch,) result will not broadcast against (batch, features).
  • Reducing the wrong axis: always write down the shape before and after.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which shapes can be added together by broadcasting?

    Aligned from the right, 4 matches 4 and the vector is stretched across the 3 rows. (3,) would be compared with 4 and fail.
  2. 2. What is the shape of tf.reduce_sum(t, axis=0) when t has shape (2, 3)?

    Axis 0 disappears, leaving the size-3 axis.
  3. 3. What does a * b do for two matrices?

    * is element-wise. Use @ or tf.matmul for matrix multiplication.
  4. 4. What is the output shape of matmul of (32, 784) and (784, 10)?

    The inner 784 cancels, leaving (32, 10).

Summary

  • Element-wise operations keep shapes; @ / tf.matmul combines the inner dimension away.
  • Reductions collapse the named axis; keepdims=True preserves it as size 1.
  • Broadcasting aligns shapes from the right and stretches size-1 axes.
  • Softmax turns logits into probabilities; subtract the max for stability.
  • Prefer vectorized operations to Python loops.

Next: Variables and Data Types