Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 62 ยท Vector Spaces

Linear Independence

Learn what linearly independent and dependent vectors are, how to test them with matrix rank in NumPy, and how to find and remove redundant vectors.

In the last lesson you saw that some vectors add nothing to a span: if v2 is just 2*v1, then span{v1, v2} is the same line as span{v1}. Vectors like that are redundant. Linear independence is the precise way to say "no vector in this collection is redundant", and linear dependence says "at least one of them can be built from the others".

Why care? Redundancy shows up everywhere: a data table with two columns that carry the same information, a system of equations with a repeated equation, or a robot with more joints than it needs. Independent vectors are the efficient building blocks of a space, and they lead directly to the idea of a basis. In this lesson you will learn the definition, three practical tests, and how to identify exactly which vectors are redundant.

The definition

Vectors v1, ..., vk are linearly independent if the only way to combine them to get the zero vector is with all weights zero:

c1*v1 + ... + ck*vk = 0 implies c1 = c2 = ... = ck = 0.

If some nonzero choice of weights also gives the zero vector, the vectors are linearly dependent. A dependence such as 2*v1 - 1*v2 = 0 can be rearranged to v2 = 2*v1: one vector is a combination of the others, which is the redundancy we wanted to capture.

Independent or dependent?

Independent Each vector adds a new direction.
Dependent One vector can be built from the others.

The zero vector alone already makes a set dependent.

1 / 2

Testing with rank

Put the vectors into the columns of a matrix V. The vectors are independent exactly when V c = 0 has only the solution c = 0, which happens exactly when every column holds a pivot, that is, when the rank of V equals the number of columns. So the test is one line: compare np.linalg.matrix_rank(V) with the number of vectors.

Python The rank test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a square matrix, a nonzero determinant is an equivalent test (see Determinants), but the rank test works for any shape and is the one to remember. Another option is to run the RREF and look for a pivot in every column.

matrix_rank uses a tolerance based on the singular values, because after rounding a column that "should" be dependent may leave a tiny nonzero remainder. Do not test independence by checking whether the determinant is exactly 0.0.

Finding the redundancy

Knowing the vectors are dependent is only half the story. Usually you also want to know how: which vector can be written in terms of the others, and with what weights? The dependency relations are the solutions of V c = 0, the null space of V. The SVD gives them directly: the right singular vectors belonging to zero singular values are the null-space directions.

Python Extracting the dependency

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The dependency V c = 0 has the form c1*v1 + c2*v2 + c3*v3 = 0. Solving for v3 gives its recipe in terms of v1 and v2, which is exactly how we built it: 2*v1 - 3*v2. (The SVD may flip the overall sign of c, but the recipe comes out the same.)

Which vectors to keep?

If you have a collection of vectors and want an independent subset that gives the same span, keep the pivot columns of the RREF. This is a standard way to remove redundancy: scan the columns from left to right and keep a vector only when it is not already in the span of those kept so far.

Python Drop the redundant vectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both the original list and the reduced list have the same rank, so they span exactly the same space, but the reduced list is lean: no vector is wasted. The size of such a lean list is the dimension of the span. That is the subject of the next lessons.

Independence in practice

Independence is not just a classroom idea.

  • Features in data. Two columns that are perfectly correlated, or one that equals the sum of others, make a design matrix rank-deficient and break ordinary linear regression.
  • Equations. A system with dependent rows has fewer real constraints than it appears, so it usually has free variables.
  • Nearly dependent vectors. Real data are rarely exactly dependent; the troublesome case is almost dependent, where the matrix is technically full rank but badly conditioned. Singular values reveal that: a tiny one means nearly dependent.
Python Exactly versus nearly dependent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The nearly dependent matrix is reported as full rank (3) because its smallest singular value is tiny but not tiny enough for the default tolerance to ignore. With a looser tolerance it counts as rank 2. Deciding which one is right depends on how noisy your data are, and you will return to this in Numerical Linear Algebra Pitfalls.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Vectors v1, v2, v3 satisfy 2v1 - v2 + 4v3 = 0. What can you conclude?

    A combination with nonzero weights gives the zero vector, which is the definition of linear dependence. Any one of them can be solved for in terms of the others.
  2. 2. A matrix V has 5 columns and np.linalg.matrix_rank(V) returns 4. What does that tell you?

    Independent columns require rank equal to the number of columns. Rank 4 with 5 columns means at least one column is redundant.
  3. 3. Is it possible for 4 vectors in R^3 to be linearly independent?

    The rank of a 3-row matrix is at most 3, so four columns always include a dependency.
  4. 4. Which set is linearly dependent?

    (-3, -6) is -3 times (1, 2), so the pair is dependent. The others are independent: the last one is a 3 by 3 matrix with a nonzero determinant.

Summary

  • Vectors are independent when only the all-zero weights combine them to zero; otherwise one of them is a combination of the others.
  • Test with rank: independent columns means matrix_rank(V) equals the number of columns, and more than n vectors in R^n are always dependent.
  • The dependency relations are the null-space solutions of V c = 0; the SVD or RREF reveals them.
  • Keeping the pivot columns gives an independent subset with the same span.
  • Exact dependence is rare in real data; watch for tiny singular values that signal near dependence.

Next: independent vectors that span a whole space form a basis, in Basis and Dimension.