Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 62 ยท Eigenvalues and Decompositions

Tensors, Kronecker Products and Einsum

Go beyond matrices: understand tensors as multi-dimensional arrays, the Kronecker product and its eigenvalue rules, the vec identity and einsum contractions in NumPy.

A vector is a list of numbers. A matrix is a grid of numbers. What comes next? A cube of numbers, a four-dimensional block of numbers, and so on. A color image is height by width by three channels. A batch of 64 such images adds a fourth axis. A video adds time. All of these are tensors: multi-dimensional arrays, the natural data structure of modern machine learning, physics and signal processing.

This lesson explains what tensors are, shows how to build and manipulate them, and introduces two tools that connect them to the matrix algebra you already know: the Kronecker product, which builds a big matrix from two small ones, and Einstein summation (einsum), a compact notation for multiplying and summing along chosen axes. It builds on matrix multiplication, eigenvalues and determinants. The NumPy side of einsum is covered in detail in Einstein summation with einsum and tensordot; here we focus on the linear-algebra meaning.

What is a tensor?

In this course a tensor of order d is an array with d indices, for example T[i, j, k] for order 3. Vectors are order 1, matrices order 2, scalars order 0. In NumPy the order is ndarray.ndim and the shape lists the length of each axis.

A warning about words: the order of a tensor (number of axes) has nothing to do with the rank of a matrix (dimension of its column space). Physicists also use "tensor" for objects that transform in a specific way under a change of basis; a tensor in the NumPy sense is just a multi-dimensional array, which is enough for most data work.

Basic operations:

  • Indexing and slicing select entries, rows, whole "slabs".
  • Reshaping reinterprets the same numbers with a different shape.
  • Outer product a โŠ— b โŠ— c of vectors builds a tensor with entries a[i]*b[j]*c[k]. A tensor that is an outer product of vectors is called rank-one (this is a different "rank" again, the tensor rank). Every tensor is a sum of rank-one tensors, and finding a short such sum (the CP decomposition) is the tensor counterpart of low-rank matrix approximation, see low-rank approximation.
  • Contraction sums over a pair of indices: trace and matrix multiplication are both contractions.
Python Tensors as arrays: shape, slices and outer products

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Kronecker product

The Kronecker product A โŠ— B of an m-by-n matrix A and a p-by-q matrix B is the (m*p)-by-(n*q) matrix made of blocks: block (i, j) is a_ij * B. For example

[[1, 2], [3, 4]] โŠ— [[0, 1], [1, 0]] is the 4-by-4 matrix [[0, 1, 0, 2], [1, 0, 2, 0], [0, 3, 0, 4], [3, 0, 4, 0]].

Think of it as "every entry of A replaced by a scaled copy of B". It is not the ordinary matrix product and does not require matching sizes. NumPy computes it with np.kron.

The algebra is surprisingly clean:

  • Mixed product: (A โŠ— B)(C โŠ— D) = (A C) โŠ— (B D), whenever the products A C and B D exist.
  • Transpose and inverse: (A โŠ— B)^T = A^T โŠ— B^T, and (A โŠ— B)^(-1) = A^(-1) โŠ— B^(-1) for invertible square matrices.
  • Eigenvalues: if A has eigenvalues lambda_i and B has eigenvalues mu_j, then A โŠ— B has the n*m eigenvalues lambda_i * mu_j, with eigenvectors v_i โŠ— w_j.
  • Determinant and trace: for A of size n and B of size m, det(A โŠ— B) = det(A)**m * det(B)**n and trace(A โŠ— B) = trace(A) * trace(B).
  • Rank: rank(A โŠ— B) = rank(A) * rank(B).

These identities let you analyze a huge matrix through its small factors. If A and B are each 100-by-100, A โŠ— B is 10,000-by-10,000, but its eigenvalues cost only two small eigen-decompositions.

Python Kronecker product properties, verified

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The vec trick

Many unknowns come as a matrix X, but linear-algebra tools want a vector. The vectorization vec(X) stacks the columns of X into one long vector (column-major order, which is NumPy's order="F"). The key identity is

vec(A @ X @ B) = (B^T โŠ— A) @ vec(X).

It turns a matrix equation such as A @ X @ B = C into an ordinary linear system (B^T โŠ— A) @ vec(X) = vec(C) that any solver can handle. The Sylvester equation A X + X B = C becomes (I โŠ— A + B^T โŠ— I) vec(X) = vec(C), and least-squares problems with matrix unknowns work the same way. The Kronecker system is big (size n*m), so for large problems you avoid forming it and use the structure directly; but it is the right way to think and to check.

Python Solving A X B = C with the vec identity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Kronecker products in practice

Separable filters. A 2-D blur kernel that factors as an outer product u v^T can be applied as two 1-D passes. In the Kronecker language, the operator on the flattened image is kron(V, U), a huge matrix you never build but whose structure gives a cheap algorithm.

Discretized PDEs. The 2-D Laplace operator on an n-by-n grid is kron(I, T) + kron(T, I), where T is the 1-D second-difference matrix (2 on the diagonal, -1 next to it). Its eigenvalues are lambda_i + lambda_j, sums of the 1-D eigenvalues. That is how one solves the Poisson problem and why fast solvers exist for it.

Quantum computing. The state of two qubits lives in the Kronecker product of two 2-dimensional spaces, and gates on separate qubits combine as U โŠ— V.

Python The 2-D Laplacian from Kronecker products

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Einsum: contractions in one line

Einstein's notation sums over repeated indices. np.einsum takes a subscript string: indices that appear in the inputs but not in the output are summed. So

  • "ij,jk->ik" is matrix multiplication,
  • "ii->" is the trace and "ij->ji" the transpose,
  • "i,j->ij" is the outer product,
  • "bij,bjk->bik" is a batch of matrix products over a leading batch axis b,
  • "ijk,k->ij" contracts a tensor with a vector along its last axis.

The Kronecker product itself is an einsum followed by a reshape: np.einsum("ij,kl->ikjl", A, B).reshape(m*p, n*q) reproduces np.kron(A, B). This shows the Kronecker product is "a four-index tensor, rearranged into a matrix".

Python einsum for common linear-algebra operations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Confusing the Kronecker product kron(A, B) with the elementwise product A * B or the matrix product A @ B.
  • Getting the order wrong in the vec identity: it is kron(B.T, A) for vec(A X B) with column-major vec. Row-major flattening (reshape(-1) in NumPy's default) changes the identity to kron(A, B.T).
  • Mixing up tensor order (number of axes), matrix rank and tensor rank (number of rank-one terms).
  • Forming kron of two large matrices when you only need to apply it; apply the factors one axis at a time.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. If A is 2 x 3 and B is 4 x 5, what is the shape of kron(A, B)?

    The Kronecker product has (24) rows and (35) columns: 8 x 15.
  2. 2. If A has eigenvalues 1 and 3, and B has eigenvalues 2 and 5, what are the eigenvalues of kron(A, B)?

    The eigenvalues of A kron B are all products lambda_i * mu_j: 12, 15, 32, 35.
  3. 3. Which einsum string computes the trace of a square matrix?

    The repeated index i on one operand sums the diagonal entries; no output indices remain, so the result is a scalar.
  4. 4. With column-major vec, vec(A X B) equals what?

    The identity vec(A X B) = (B^T kron A) vec(X) turns matrix equations into ordinary linear systems.
  5. 5. A color video clip has axes (time, height, width, channel). What is its tensor order?

    The order is the number of axes (indices), which is 4 here.

Summary

  • A tensor is an array with several indices; order = number of axes, shape = their lengths. Do not confuse order with matrix rank.
  • The Kronecker product A โŠ— B places a scaled copy a_ij * B in each block; it obeys (A โŠ— B)(C โŠ— D) = (AC) โŠ— (BD) and its eigenvalues are all products lambda_i * mu_j.
  • det(A โŠ— B) = det(A)**m * det(B)**n, trace(A โŠ— B) = trace(A) * trace(B), and ranks multiply.
  • vec(A X B) = (B^T โŠ— A) vec(X) turns matrix equations (including Sylvester equations) into ordinary linear systems.
  • einsum expresses contractions (matmul, trace, batch products, quadratic forms) compactly, and a Kronecker product is an einsum plus a reshape.

Next: put everything together with the PCA project, or test yourself with the eigenvalue and SVD exercises.