Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 40 ยท Arrays and NumPy API

Reductions, Linear Algebra and Math Functions

Use jax.numpy for reductions along axes, cumulative ops, sorting, matrix products, jnp.linalg solves and decompositions, einsum, and jax.scipy special functions.

Most numerical code is built from a few families of operations: reductions (sum, mean, max), element-wise math, matrix multiplication, and linear-algebra routines. jax.numpy offers all of them with NumPy's names and axis conventions, and each one is differentiable and compilable. This lesson is a guided tour.

Runnable editors use NumPy, whose calls are identical to the JAX versions shown in the read-only blocks (JAX cannot run in the in-browser runtime; see What Is JAX?). Printed values are float64 in NumPy; in JAX you would see float32 and tiny last-digit differences.

Reductions along axes

A reduction collapses one or more axes into a single value per remaining position. The axis argument says which axis disappears.

Python
import jax.numpy as jnp

a = jnp.array([[1., 2., 3.],
               [4., 5., 6.]])

jnp.sum(a)              # 21.0
jnp.sum(a, axis=0)      # [5. 7. 9.]   collapses rows -> one value per column
jnp.sum(a, axis=1)      # [ 6. 15.]    collapses columns -> one value per row
jnp.mean(a, axis=1, keepdims=True)   # shape (2, 1), handy for broadcasting back

Other reductions: prod, min, max, argmin, argmax, std, var, any, all, median, percentile, logsumexp (in jax.scipy.special). keepdims=True keeps the reduced axis with size 1, which makes the result broadcastable against the original (broadcasting).

Python Reductions and keepdims (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last example shows a classic pattern: shifting by the maximum avoids exp overflow. JAX ships jax.nn.softmax and jax.scipy.special.logsumexp that already do this.

Cumulative operations, sorting and searching

Python
x = jnp.array([3., 1., 4., 1., 5.])
jnp.cumsum(x)               # [ 3.  4.  8.  9. 14.]
jnp.cumprod(x)
jnp.sort(x)                 # [1. 1. 3. 4. 5.]
jnp.argsort(x)              # indexes that would sort x
jnp.searchsorted(jnp.sort(x), 3.5)   # insertion position
jnp.unique(x)               # needs size= under jit, see below

jnp.unique, jnp.nonzero, jnp.argwhere and similar functions produce results whose length depends on the data. Outside jit they work as usual. Inside jit you must pass a fixed size= (and optionally fill_value=), as covered in jit pitfalls.

Matrix products

The @ operator and jnp.matmul do matrix multiplication; jnp.dot and jnp.vdot handle the vector cases; jnp.einsum is a general tool.

Python
A = jnp.ones((3, 4))
B = jnp.ones((4, 5))
C = A @ B                      # (3, 5)

# batched matmul: leading axes are batch dimensions
X = jnp.ones((8, 3, 4))
Y = jnp.ones((8, 4, 5))
Z = X @ Y                      # (8, 3, 5)

On GPU and TPU, matrix multiplication is where the speed comes from. JAX lets you trade precision for speed with precision= (jax.lax.Precision), because TPUs and some GPUs use reduced precision for float32 multiplies by default.

einsum describes a computation by naming each axis with a letter. Letters that appear in the inputs but not in the output are summed over:

Python einsum recipes (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

jnp.einsum mirrors this exactly, and XLA turns it into efficient matrix multiplications. Use it for attention, tensor contractions and anywhere shapes get hard to follow.

Linear algebra: jnp.linalg

Python
M = jnp.array([[2., 1.], [1., 3.]])
b = jnp.array([1., 2.])

jnp.linalg.solve(M, b)       # solve M x = b
jnp.linalg.inv(M)            # inverse (prefer solve when possible)
jnp.linalg.det(M)            # 5.0
jnp.linalg.eigh(M)           # eigenvalues/vectors of a symmetric matrix
jnp.linalg.svd(M)            # singular value decomposition
jnp.linalg.norm(b)           # Euclidean length
jnp.linalg.cholesky(M)       # for symmetric positive-definite matrices
jnp.linalg.qr(M)
jnp.linalg.lstsq(M, b)       # least squares

All of these can be differentiated by grad (for example, the gradient of the log-determinant), and they run on the accelerator. jax.scipy.linalg adds more (lu, expm, triangular solves), and jax.scipy.special offers gamma, erf, logsumexp, expit, and friends.

Python Solving and decomposing (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Element-wise math and a few gotchas

jnp.exp, log, sin, cos, tanh, sqrt, power, clip, abs, maximum, where and the rest apply element-wise and broadcast. Two cautions:

  • NaNs propagate and there are no warnings on accelerators: jnp.log(-1.0) is nan. Debug with jax.config.update("jax_debug_nans", True), which raises an error where the first NaN appears.
  • Division by zero gives inf or nan, never an exception. This also affects gradients: the derivative of sqrt at 0 is inf. See the grad lessons.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. For a (2, 3) array a, what is the shape of jnp.sum(a, axis=0)?

    axis=0 collapses the rows, leaving one value per column: shape (3,).
  2. 2. What does keepdims=True do?

    The reduced axis stays as length 1, which is convenient for operations like subtracting a row mean.
  3. 3. Which einsum string computes the matrix product of A (i, j) and B (j, k)?

    The shared letter j is summed over, leaving the output axes i and k.
  4. 4. Why use jnp.linalg.solve(M, b) instead of jnp.linalg.inv(M) @ b?

    Solving directly avoids forming the inverse, which is slower and numerically less stable.

Summary

  • Reductions take an axis; keepdims=True helps broadcast the result back.
  • Cumulative, sorting and search functions mirror NumPy; data-dependent output sizes need size= under jit.
  • @, matmul and einsum express matrix products and tensor contractions; XLA optimizes them.
  • jnp.linalg and jax.scipy provide solves and decompositions, all differentiable.
  • NaNs and infinities propagate silently; jax_debug_nans helps find the source.

Next: Low-level Operations and Control Flow with lax