Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 62 · Determinants and Orthogonality

Orthogonality and Projections

Learn orthogonal vectors and subspaces, project a vector onto a line or a subspace with the normal equations, and verify the projection matrix properties in NumPy.

Perpendicular directions are the friendliest ones in linear algebra. Measurements along perpendicular axes do not interfere with each other, lengths follow the Pythagorean theorem, and many calculations that need a matrix inverse collapse into simple dot products. Projection puts that to work: it finds the closest point to a vector inside a line, a plane or any subspace.

This lesson covers orthogonal vectors and subspaces, then derives projection onto a line, generalizes it to a subspace through the normal equations, and checks the properties of projection matrices. The result is the foundation for Gram-Schmidt and least squares.

Why perpendicular is nice

Independent Perpendicular axes do not mix.
Pythagoras Squared lengths simply add.
Dot product 0 An easy test for any dimension.
1 / 2

Orthogonal vectors

Two vectors are orthogonal when their dot product is zero. In 2D and 3D that is the same as perpendicular. The zero vector is orthogonal to everything, and the definition works in any number of dimensions where you cannot draw a picture. The reason it matches the geometry comes from the formula a·b = |a||b|cos θ in The Dot Product: zero dot product means cos θ = 0.

Orthogonality has a Pythagorean consequence. If a·b = 0, then |a + b|² = |a|² + |b|².

Python Orthogonal vectors and Pythagoras

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A set of nonzero vectors that are all orthogonal to each other is automatically linearly independent. If you additionally scale each to length 1, the set is orthonormal. Matrices with orthonormal columns satisfy QᵀQ = I, which you will meet again in the QR lesson.

Orthogonal subspaces

Two subspaces V and W are orthogonal if every vector in V is orthogonal to every vector in W. The most important example comes from the four subspaces of a matrix (Column Space and Null Space):

  • The null space of A is orthogonal to the row space of A. A vector x in the null space satisfies Ax = 0, which says exactly that every row of A has zero dot product with x.
  • The left null space (null space of Aᵀ) is orthogonal to the column space.

The orthogonal complement of a subspace is the set of all vectors orthogonal to it. In R³ the complement of a plane through the origin is the line perpendicular to it.

Python Null space is orthogonal to the row space

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Projecting onto a line

Given a vector b and a line through the origin in the direction of a, which point on the line is closest to b? Call it p = x̂ a. The error e = b − p must be perpendicular to the line, so a·(b − x̂ a) = 0. Solving for the scalar:

TEXT
x̂ = (a·b) / (a·a)        p = x̂ a

If a has length 1, a·a = 1 and the formula is just p = (a·b) a. As a matrix: P = a aᵀ / (aᵀa), so p = P b.

Python Project a vector onto a line

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here a·b = 5 and a·a = 5, so x̂ = 1: the projection of (1, 3) is (2, 1) and the error is (−1, 2). The error is perpendicular to a, which is the property that matters. Try other b values to see the pattern.

Projecting onto a subspace

Now the target is a subspace spanned by the columns of a matrix A (the columns must be independent). We look for the point p = A x̂ in the column space that is closest to b. The error e = b − A x̂ must be orthogonal to every column of A, which means Aᵀ e = 0:

TEXT
Aᵀ (b − A x̂) = 0      →      AᵀA x̂ = Aᵀb

These are the normal equations. When the columns are independent, AᵀA is invertible, so

TEXT
x̂ = (AᵀA)⁻¹ Aᵀ b        p = A x̂        P = A (AᵀA)⁻¹ Aᵀ

Projection onto a line is the special case where A has a single column.

Python Projection onto a plane with the normal equations

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The closest point in the plane is p = (5, 2, −1) and the error is e = (1, −2, 1). The vector b splits into a part inside the subspace and a part orthogonal to it, and the two parts obey the Pythagorean theorem.

Properties of projection matrices

The matrix P = A(AᵀA)⁻¹Aᵀ has two defining properties:

  • Symmetric: Pᵀ = P.
  • Idempotent: P² = P. Projecting a second time changes nothing, because the first projection already landed inside the subspace.

Also, I − P projects onto the orthogonal complement, so b = Pb + (I − P)b splits any vector into its two perpendicular pieces. The eigenvalues of a projection matrix are only 0 and 1.

Python Checking the projection matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In real code, avoid forming (AᵀA)⁻¹. Solve the normal equations with np.linalg.solve, or better, use np.linalg.lstsq or a QR factorization, which are more accurate when the columns are nearly dependent. The QR lesson shows why.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. Which pair of vectors is orthogonal?

    (1)(-2) + (2)(1) = 0. The other pairs have dot products 4, 5 and 3.
  2. 2. What is the projection of b onto the line through a?

    The scalar (a . b)/(a . a) makes the error b - p perpendicular to a; multiplying it by a gives the point on the line.
  3. 3. What do the normal equations A^T A x = A^T b say geometrically?

    A^T (b - A x) = 0 means each column of A has zero dot product with the error, so the error is perpendicular to the column space.
  4. 4. Which statement about a projection matrix P is true?

    Projecting twice is the same as projecting once, so P^2 = P. Unless P is the identity, it is not invertible, because it collapses the orthogonal part to zero.

Summary

  • Vectors are orthogonal when their dot product is zero; orthogonal nonzero vectors are independent and obey the Pythagorean theorem.
  • The null space is orthogonal to the row space, and the left null space to the column space.
  • Projection onto a line: p = ((a·b)/(a·a)) a.
  • Projection onto a column space: solve AᵀA x̂ = Aᵀb, then p = A x̂; the error is orthogonal to the subspace.
  • Projection matrices are symmetric and satisfy P² = P; I − P projects onto the orthogonal complement.

Next up: Gram-Schmidt and QR Decomposition, which turns any basis into an orthonormal one.