Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 49 ยท Linear Algebra

Einstein Summation with einsum and tensordot

Learn NumPy einsum notation for matrix multiply, batch matmul, trace and outer products, plus tensordot for higher-dimensional arrays and the optimize flag.

np.einsum expresses almost any combination of multiplying and summing arrays in one short string. Instead of remembering a different function for dot products, matrix products, traces, outer products and batched operations, you describe the operation by naming the axes. Once the notation clicks, you can read and write tensor code in a single line, and many operations avoid temporary arrays.

The idea comes from physics: when an index letter appears in the input but not in the output, it is summed over. That one rule explains everything. You should already know @, np.dot and np.outer from Vectors, Dot Products and the @ Operator, and broadcasting will help you appreciate what einsum replaces.

The notation

An einsum call looks like np.einsum("ij,jk->ik", A, B). The string has three parts:

  • Before the ->: one group of letters per input array, separated by commas. Each letter names one axis of that array.
  • After the ->: the letters of the output.
  • Rule 1: a letter that appears in more than one input is multiplied along that axis (the axes must have equal length).
  • Rule 2: a letter that is missing from the output is summed over.

In "ij,jk->ik", j appears in both inputs (multiply) and is absent from the output (sum). That is exactly matrix multiplication.

flowchart LR
    A["A: letters 'ij'"] --> M["multiply where letters match (j)"]
    B["B: letters 'jk'"] --> M
    M --> S["sum letters missing from the output (j)"]
    S --> O["output: letters 'ik'"]

einsum multiplies along shared letters, then sums the letters that are dropped from the output.

Common operations in einsum

Python Familiar operations written as einsum

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read each string aloud: "ij->j" means "keep j, sum i", so it sums down the rows to give column sums. "ij->ji" keeps both letters but swaps their order, which is a transpose.

Diagonal and trace

Repeating a letter in a single input picks the diagonal: "ii->i" extracts it, and "ii->" sums it, which is the trace.

Python Diagonal and trace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line shows a real benefit: the trace of A @ B is computed without ever forming the full product.

Batches of matrices

Matrix multiplication of 3D arrays treats the first axis as a batch of independent matrices. In einsum, add a letter that appears in every input and in the output, and it simply rides along:

np.einsum("bij,bjk->bik", A, B) multiplies matrix b of A with matrix b of B, for every b. This is the same as A @ B, which also broadcasts over leading axes.

Python Batch matrix multiplication

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

flowchart TD
    A["A: shape (b, i, j)"] --> E["'bij,bjk->bik'"]
    B["B: shape (b, j, k)"] --> E
    E --> C["C: shape (b, i, k)"]

The batch letter b is kept in the output, so each pair of matrices is multiplied independently.

Ellipsis for any number of leading axes

Writing ... stands for "all other axes". "...ij,...jk->...ik" works for any number of batch dimensions, including none. This is how library code stays flexible.

Python Using ... in einsum

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A realistic use: attention-style scores

Many machine-learning formulas are awkward with @ but natural with einsum. Here is how similar each query vector is to each key vector, for several batch items at once:

Python Scores between queries and keys

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

tensordot

np.tensordot(a, b, axes=...) sums products over chosen axes, and everything left over is concatenated into the output shape. With an integer, axes=n contracts the last n axes of a with the first n of b. With a pair of lists you pick the axes by number.

Python tensordot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use tensordot when you can name the contracted axes by number; use einsum when you want the output order spelled out. They are interchangeable for most work.

The optimize flag

With three or more operands, the order in which pairs are combined changes the cost enormously. By default einsum evaluates naively in one big loop. Pass optimize=True (or "optimal") to let NumPy pick a good order, often using tensordot or BLAS routines behind the scenes. np.einsum_path shows the plan and the estimated speedup.

Python Choosing a contraction order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Results match, but the optimized route multiplies pairs in sequence instead of looping over all four indexes at once. As a rule of thumb, add optimize=True for three or more operands, and keep two-operand calls simple.

When a formula is a plain matrix product, A @ B is clearer and just as fast. Reach for einsum when you need to permute axes, take diagonals, or contract over unusual axes, not as a replacement for @.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In np.einsum("ij,jk->ik", A, B), what happens to the letter j?

    A letter shared by the inputs is multiplied along that axis, and because it is missing from the output it is summed. That is matrix multiplication.
  2. 2. Which string computes the trace of a square matrix?

    Repeating a letter in one input selects the diagonal, and leaving it out of the output sums it. 'ii->i' returns the diagonal itself.
  3. 3. What does the string "bij,bjk->bik" compute?

    The letter b appears in both inputs and in the output, so it is a batch axis that rides along while i, j and k form each matrix product.
  4. 4. When is optimize=True most useful?

    The contraction order can change the cost by orders of magnitude. optimize lets NumPy choose a good order.

Summary

  • einsum names every axis with a letter: shared letters are multiplied, letters dropped from the output are summed.
  • "i,i->" is a dot product, "i,j->ij" an outer product, "ij,jk->ik" a matrix product, "ii->" a trace, "ij->ji" a transpose.
  • A letter present in all inputs and the output is a batch axis; ... covers any number of leading axes.
  • np.tensordot contracts chosen axes numerically; axes=0 gives an outer product.
  • Use optimize=True for three or more operands, and np.einsum_path to inspect the plan.
  • Prefer @ for plain matrix products; use einsum for unusual axis patterns.

The linear algebra section is complete. Next, move on to randomness with Random Numbers with Generators.