Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 62 · Determinants and Orthogonality

Inner Product Spaces, Norms and Cauchy-Schwarz

Generalize the dot product to inner products on matrices and functions, learn the norm axioms, and prove and test Cauchy-Schwarz and the triangle inequality.

The dot product gives arrows in the plane a length and an angle: u · v = |u||v| cos θ. Those two ideas, length and angle, are all you need to talk about orthogonality, projection and least squares. The remarkable thing is that they do not depend on the vectors being lists of numbers. Any vector space that has an operation behaving like the dot product gets the same geometry for free. That operation is called an inner product.

This lesson states what an inner product must satisfy, shows examples on matrices and functions, defines norms from inner products, and proves the single most useful inequality in the subject, Cauchy-Schwarz. Together with the vector space axioms this is the foundation under orthogonality, Gram-Schmidt and Fourier series.

What an inner product is

An inner product on a real vector space assigns to every pair of vectors u, v a real number ⟨u, v⟩ such that for all vectors and scalars:

  1. Symmetry: ⟨u, v⟩ = ⟨v, u⟩.
  2. Linearity: ⟨au + bw, v⟩ = a⟨u, v⟩ + b⟨w, v⟩.
  3. Positive definiteness: ⟨v, v⟩ ≥ 0, and ⟨v, v⟩ = 0 only when v = 0.

(For complex spaces, symmetry becomes ⟨u, v⟩ = conj(⟨v, u⟩); see complex matrices.) A vector space with an inner product is an inner product space. The inner product defines:

  • the norm (length): ‖v‖ = sqrt(⟨v, v⟩);
  • the distance: d(u, v) = ‖u - v‖;
  • orthogonality: u ⊥ v when ⟨u, v⟩ = 0;
  • the angle: cos θ = ⟨u, v⟩ / (‖u‖ ‖v‖).

Examples

  • R^n with the dot product: ⟨x, y⟩ = Σ x_i y_i. The standard one.
  • Weighted dot product: ⟨x, y⟩ = Σ w_i x_i y_i with positive weights w_i. It makes some coordinates count more than others. More generally, ⟨x, y⟩ = xᵀ A y is an inner product exactly when A is symmetric positive definite (as in the Cholesky lesson).
  • Matrices with the Frobenius inner product: ⟨A, B⟩ = trace(AᵀB) = Σ A_ij B_ij. Matrices become vectors with m · n entries.
  • Functions on an interval: ⟨f, g⟩ = ∫ f(x) g(x) dx over [a, b]. The functions sin x and cos x are orthogonal on [0, 2π], which underlies Fourier series.

One idea, many spaces

Arrows The ordinary dot product.
Matrices Frobenius product, trace of A^T B.
Functions Integral of f times g.
1 / 2

Norms and their axioms

A norm on a vector space is a function ‖·‖ that assigns each vector a length and satisfies:

  1. Positive definite: ‖v‖ ≥ 0, and ‖v‖ = 0 only for v = 0.
  2. Homogeneous: ‖c v‖ = |c| ‖v‖.
  3. Triangle inequality: ‖u + v‖ ≤ ‖u‖ + ‖v‖.

Every inner product produces a norm. Not every norm comes from an inner product: the 1-norm Σ|x_i| and the max-norm max|x_i| satisfy the axioms but are not built from any inner product. A quick test is the parallelogram law ‖u + v‖² + ‖u - v‖² = 2‖u‖² + 2‖v‖², which holds exactly for norms that come from an inner product.

The Cauchy-Schwarz inequality

For any two vectors in an inner product space,

TEXT
|⟨u, v⟩| ≤ ‖u‖ · ‖v‖

with equality exactly when one vector is a multiple of the other. It is what makes "cosine of the angle" meaningful, because it guarantees -1 ≤ cos θ ≤ 1.

Proof. If v = 0 both sides are zero. Otherwise, subtract from u its projection onto v: let w = u - (⟨u, v⟩ / ⟨v, v⟩) v. Then w is orthogonal to v, and by positive definiteness

TEXT
0 ≤ ⟨w, w⟩ = ⟨u, u⟩ - ⟨u, v⟩² / ⟨v, v⟩

Multiply by ⟨v, v⟩ to get ⟨u, v⟩² ≤ ⟨u, u⟩⟨v, v⟩, and take square roots. Equality needs w = 0, meaning u is a multiple of v. ∎

The triangle inequality follows: ‖u + v‖² = ‖u‖² + 2⟨u, v⟩ + ‖v‖² ≤ ‖u‖² + 2‖u‖‖v‖ + ‖v‖² = (‖u‖ + ‖v‖)².

Python Three inner products, the norm axioms and Cauchy-Schwarz

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weighted product is a legitimate inner product, and the integral inner product reproduces the classic facts about sine and cosine. The last loop shows that only the 2-norm obeys the parallelogram law for these vectors, a numerical hint that the 1-norm and max-norm do not come from an inner product.

Why it matters

  • Projection and least squares work in any inner product space: the best approximation of v in a subspace is the vector whose error is orthogonal to the subspace. With the integral inner product this becomes best polynomial or Fourier approximation of a function.
  • Cosine similarity, used in search engines and recommenders, is ⟨u, v⟩/(‖u‖‖v‖), and Cauchy-Schwarz guarantees it stays between -1 and 1.
  • Covariance cov(X, Y) is an inner product on centered random variables, so the correlation cov/(σ_X σ_Y) lies in [-1, 1] for the same reason.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. Which of these must an inner product satisfy?

    Positive definiteness requires <v, v> > 0 for nonzero v. Linearity says <2u, v> = 2<u, v>, not <u, v> + 2.
  2. 2. State the Cauchy-Schwarz inequality.

    The absolute inner product never exceeds the product of the norms, with equality exactly for parallel vectors.
  3. 3. For u = (1, 2) and v = (3, 4), which quantity is larger: |u . v| or ||u|| ||v||?

    u . v = 11 and the product of norms is about 11.18, consistent with Cauchy-Schwarz.
  4. 4. The functions sin x and cos x on [0, 2pi] with <f, g> = integral of fg are...

    The integral of sin x cos x over a full period is 0, so the two functions are orthogonal.
  5. 5. Which norm does NOT come from an inner product?

    The 1-norm fails the parallelogram law, so no inner product produces it.

Summary

  • An inner product is symmetric, linear in each argument and positive definite; it defines norm, distance, angle and orthogonality in any vector space.
  • Examples: the dot product, xᵀAy with A symmetric positive definite, the Frobenius product trace(AᵀB), and ∫ f g dx for functions.
  • A norm is positive definite, homogeneous and obeys the triangle inequality; only some norms come from inner products (the parallelogram law tells which).
  • Cauchy-Schwarz, |⟨u, v⟩| ≤ ‖u‖‖v‖, is proved by subtracting a projection, and it implies the triangle inequality and the meaning of "cosine".

Next: Orthogonality and Projections, where inner products measure and split vectors.