Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 49 ยท Linear Algebra

Vectors, Dot Products and the @ Operator

Learn vectors in NumPy with dot, vdot, inner, outer, matmul and the @ operator, matrix shape rules, norms, the angle between vectors and the cross product.

Linear algebra is the language of graphics, physics, statistics and machine learning, and NumPy is where most Python programmers practice it. You do not need a math degree to start. A vector is just a 1D array, a matrix is a 2D array, and a handful of operations on them (dot product, matrix product, norm, angle) cover a surprising share of real work: scoring a shopping basket, comparing documents, rotating a point, running a layer of a neural network.

The most important lesson in this chapter is the difference between * and @. The star multiplies element by element; the @ operator performs true matrix multiplication. We start with vectors and the dot product, then see how the rules for shapes make matrix products work, and finish with norms, angles and the cross product.

The dot product of two vectors

The dot product multiplies two vectors element by element and adds up the results:

TEXT
[1, 2, 3] . [4, 5, 6]  =  1*4 + 2*5 + 3*6  =  32

The result is a single number. It is large when the vectors point in a similar direction and zero when they are perpendicular. In NumPy there are several spellings that agree for 1D arrays:

  • u @ w (the modern, preferred operator)
  • np.dot(u, w)
  • np.inner(u, w)
  • (u * w).sum() (the by-hand version)

Two related functions are worth knowing. np.vdot(a, b) flattens its inputs and takes the complex conjugate of the first argument, which is the mathematically correct dot product for complex vectors. np.outer(a, b) goes the other way and returns a table of all pairwise products, with a down the rows and b across the columns.

Python Dot, inner, vdot and outer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shopping basket shows why the dot product is everywhere: whenever you compute "a weighted sum", you are taking a dot product. Prices times quantities, grades times weights, features times coefficients.

For complex numbers, np.dot does not conjugate and np.vdot does, which is why they print different results above. For real numbers, which is nearly all data, they agree.

Matrix multiplication and shapes

Multiplying matrices is a dot product between every row of the left matrix and every column of the right one. The entry in row i, column j of the result is the dot product of row i of A with column j of B.

That only works when the row length of A equals the column length of B. This gives the one rule worth memorizing:

(m, n) @ (n, p) gives (m, p). The two inner numbers must match, and they disappear.

flowchart LR
    A["A<br/>shape (2, 3)"] --> M{"inner sizes<br/>3 and 3 match?"}
    B["B<br/>shape (3, 2)"] --> M
    M -- "yes" --> R["A @ B<br/>shape (2, 2)"]
    M -- "no" --> E["ValueError"]

Matrix multiplication works only when the inner dimensions agree; the outer dimensions form the result's shape.

Unlike ordinary multiplication, matrix multiplication is not commutative: A @ B and B @ A usually differ, and one of them may not even exist.

Python Matrix product shapes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Several rules are visible in that output:

  • Matrix times vector. A 1D vector on the right is treated as a column and the result is 1D. If you want to keep a column shape (2, 1), reshape the vector first (see Reshaping Arrays).
  • A * v is not A @ v. The star broadcasts v across the rows and multiplies elementwise, producing a matrix of the same shape as A. This is the most common beginner mistake in this area.
  • Identity. Multiplying by the identity matrix np.eye(n) changes nothing.
  • Order matters. M @ N swaps the columns of M, while N @ M swaps its rows.
  • Stacks of matrices. With more than two dimensions, @ treats the leading dimensions as a batch and broadcasts them. (5, 2, 3) @ (3, 4) multiplies each of the five matrices by the same 3 x 4 matrix. Batch operations are covered again in Einstein Summation with einsum and tensordot.

Note: Prefer @ over np.dot for matrices. np.dot has special rules for arrays with more than two dimensions that are easy to misremember, while @ (np.matmul) always means "batch of matrix products". Also, @ does not allow multiplying by a plain scalar: use * for that.

Norms: the length of a vector

The norm of a vector measures its length. The usual one is the Euclidean or L2 norm, the square root of the sum of squares. It is the Pythagorean theorem in any number of dimensions. np.linalg.norm(v) computes it, and it takes a second argument for other norms: 1 for the sum of absolute values, np.inf for the largest absolute value.

Dividing a vector by its own norm gives a unit vector, which keeps the direction and has length 1.

For a matrix, np.linalg.norm(M) is the Frobenius norm (the same formula applied to all entries, as if the matrix were flattened). With axis= you can compute the norm of every row or column at once, and keepdims=True makes the result easy to divide with.

The angle between vectors

Rearranging the geometric definition of the dot product gives:

TEXT
cos(angle) = (a . b) / (|a| * |b|)

So the angle is arccos of the dot product divided by both lengths. The cosine itself, called cosine similarity, is widely used to compare documents or embeddings, because it ignores the vectors' lengths and looks only at direction.

Python Norms, unit vectors and angles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice two practical details. First, np.clip(c, -1, 1) guards against rounding errors: for nearly parallel vectors the computed cosine can come out as 1.0000000000000002, and arccos of that is nan. Second, the similarity table at the end is just one matrix product: normalize every row to length 1, then unit_rows @ unit_rows.T holds the cosine between every pair of documents. Documents 0 and 1 have the same direction (their counts differ only by a factor of 2), so their similarity is 1, and document 2 shares nothing with them.

The cross product

In three dimensions, np.cross(a, b) returns a new vector that is perpendicular to both inputs. Its length equals the area of the parallelogram spanned by a and b. Unlike the dot product, the result is a vector, and the order matters: cross(a, b) == -cross(b, a). The cross product of parallel vectors is the zero vector.

Warning: Passing 2D vectors (length 2) to np.cross is deprecated in NumPy 2.0. For two components, compute the single z value yourself: a[0] * b[1] - a[1] * b[0]. It is the signed area of the parallelogram, positive when b is counter-clockwise from a.

Python Cross product, areas and projection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two zeros after c = ... confirm that c is perpendicular to both a and b (a dot product of 0 means perpendicular). The last block uses the dot product to project one vector onto another, a building block of least squares, which you meet in Solving Linear Systems and Least Squares.

Inner, outer, vdot, tensordot and the pseudo-inverse

np.dot and @ cover most products, but NumPy has a few more named ones. Knowing what each does, and how it differs, prevents guessing.

Function What it computes
np.inner(a, b) sum of products over the last axis of both inputs
np.outer(a, b) every product a[i] * b[j], a matrix of shape (len(a), len(b))
np.vdot(a, b) dot product of the flattened inputs; conjugates the first argument for complex numbers
np.tensordot(a, b, axes) sum products over chosen axes; any number of dimensions
np.kron(a, b) Kronecker product, covered in matrices
np.linalg.pinv(a) Moore-Penrose pseudo-inverse of any matrix, even non-square or singular
Python inner, outer, vdot and tensordot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

np.outer is how you build a rank-one matrix from two vectors, and it is the core of covariance and attention computations. For any contraction that does not fit these functions, einsum is the general tool.

The pseudo-inverse

A normal inverse exists only for square, non-singular matrices. The pseudo-inverse np.linalg.pinv(A) exists for every matrix. For a tall matrix with independent columns it gives the least-squares solution of A x = b as pinv(A) @ b. For a singular matrix it returns the minimum-norm best answer. It is built from the singular value decomposition, which you meet in Eigenvalues and SVD.

Python pinv for non-square and singular matrices

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Prefer np.linalg.lstsq when you only need a least-squares solution (it avoids forming the pseudo-inverse), and np.linalg.solve for square systems. Use pinv when you need the matrix itself, for example to reuse it on many right-hand sides.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the result of np.array([1, 2, 3]) @ np.array([4, 5, 6])?

    For two 1D vectors @ is the dot product: 14 + 25 + 3*6 = 32. The array [4, 10, 18] is what the * operator gives.
  2. 2. A has shape (4, 3) and B has shape (3, 5). What is the shape of A @ B?

    The inner dimensions (3 and 3) match and disappear, leaving the outer dimensions (4, 5).
  3. 3. What is the difference between A * B and A @ B for two matrices?

    The star broadcasts and multiplies element by element. The @ operator takes dot products of rows with columns.
  4. 4. Why does the angle function clip the cosine to the range -1 to 1 before calling arccos?

    Floating-point rounding can give 1.0000000000000002 for parallel vectors, and arccos of a value outside [-1, 1] is nan.
  5. 5. What does np.cross(a, b) return for two 3D vectors?

    The cross product is a new vector perpendicular to a and b, with length equal to the area of the parallelogram they span.

Summary

  • The dot product is a weighted sum: u @ w equals (u * w).sum(). np.vdot conjugates complex input, np.outer builds all pairwise products.
  • * multiplies elementwise; @ is matrix multiplication. (m, n) @ (n, p) gives (m, p), the order matters, and extra leading dimensions are treated as batches.
  • np.linalg.norm gives lengths (L2 by default; axis and keepdims work for rows and columns). Dividing by the norm gives a unit vector.
  • The angle between vectors comes from arccos(a @ b / (|a| |b|)); clip the cosine first. Cosine similarity of all pairs is one matrix product of normalized rows.
  • np.cross works for 3D vectors; compute the 2D version by hand.

Next, go beyond vectors in Matrix Operations.