Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 49 ยท Linear Algebra

Solving Linear Systems and Least Squares

Solve linear systems with np.linalg.solve, fit lines with lstsq and polyfit, check conditioning with cond and handle singular matrix errors with NumPy.

A linear system is a set of equations like 2x + y = 5 and x + 3y = 10. Written with matrices it becomes A @ x = b, where A holds the coefficients, x the unknowns and b the results. Solving such systems is one of the most common jobs in scientific computing, from circuit analysis to fitting a trend line through data.

NumPy has two main tools. np.linalg.solve handles the case with exactly as many independent equations as unknowns. np.linalg.lstsq handles the messier, far more common case in data work: more equations than unknowns, where no exact answer exists and you want the best one. As the matrices lesson warned, neither uses inv.

Exact solutions with solve

np.linalg.solve(A, b) needs a square, non-singular A. It returns x such that A @ x equals b. The right-hand side b can be a vector (one system) or a matrix whose columns are several right-hand sides, solved together.

Python Solving a 3 x 3 system

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

flowchart LR
    A["A (n x n)"] --> S["np.linalg.solve"]
    B["b (n,) or (n, k)"] --> S
    S --> X["x with A @ x = b"]

solve takes a square matrix and one or more right-hand sides, and returns the unknowns.

Singular matrices

If the equations are redundant or contradictory, A is singular and solve raises LinAlgError: Singular matrix. This is useful: it tells you the problem has no unique answer.

Python Handling a singular matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Conditioning: how much can you trust the answer?

Some matrices are almost singular. They have no error, but tiny changes in the input produce huge changes in the result. The condition number, np.linalg.cond(A), measures this. Roughly, you lose log10(cond) digits of accuracy. A condition number near 1 is excellent; one around 1e16 means the double-precision result is nearly meaningless.

Python A well-conditioned and an ill-conditioned system

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The nearly parallel lines meet at a point that slides far away when b moves by one part in ten thousand. When cond is huge, think about whether your model is poorly posed, not about a better algorithm.

Least squares: best fit when there is no exact answer

Imagine measuring something ten times and wanting a straight line through the data. Ten equations, two unknowns (slope and intercept), and noise means no line passes through every point. Least squares picks the line that minimizes the sum of squared vertical gaps, called residuals.

np.linalg.lstsq(A, b, rcond=None) returns four things: the solution, the sum of squared residuals, the rank of A and its singular values. Pass rcond=None to use the modern default cutoff.

To fit y = m*x + c, build a design matrix whose first column is x and second is all ones:

Python Fitting a line with lstsq

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

flowchart TD
    D["Data points (x, y)"] --> M["Build design matrix A<br/>columns: x and 1"]
    M --> L["np.linalg.lstsq(A, y)"]
    L --> P["Parameters: slope and intercept"]
    L --> R["Residuals: how far points miss the line"]

Fitting a line is a least-squares problem on a design matrix.

The design-matrix idea is general. For a parabola use columns [x**2, x, 1]; for a model with several inputs, one column per input plus a column of ones.

polyfit and polyval

For polynomial fits there is a shortcut. np.polyfit(x, y, deg) returns the coefficients, highest power first, and np.polyval(coeffs, x) evaluates them. Under the hood polyfit builds a design matrix and runs least squares for you.

Python Polynomial fitting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A higher degree always fits the training points at least as well, but a wiggly curve through noisy data predicts new points badly. Use the lowest degree that captures the real shape. Very high degrees are also badly conditioned, so polyfit may emit a RankWarning.

Reading residuals

The residual of each point is y - A @ x. Plotting or summarizing them shows whether the fit is reasonable: they should look like random noise centered on zero. A visible pattern means the model is missing something.

Python Residual analysis

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the model includes a constant column, the residuals of a least-squares fit always average to (almost exactly) zero, so that alone does not prove the fit is good. Look at their spread and pattern.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When should you use np.linalg.lstsq instead of np.linalg.solve?

    lstsq finds the solution that minimizes the squared residuals, which is what you want for overdetermined systems such as noisy data.
  2. 2. What does a very large np.linalg.cond(A) indicate?

    The condition number measures sensitivity. A huge value means you lose many digits of accuracy when solving.
  3. 3. In np.polyfit(x, y, 2), in which order are the returned coefficients?

    polyfit and polyval use highest power first, so [a, b, c] means ax**2 + bx + c.
  4. 4. What happens when you call np.linalg.solve on a singular matrix?

    solve requires a unique solution and raises LinAlgError: Singular matrix. lstsq is the tool that still returns an answer.

Summary

  • A linear system A @ x = b is solved with np.linalg.solve(A, b) when A is square and non-singular.
  • Singular matrices raise LinAlgError; check np.linalg.matrix_rank first if unsure.
  • np.linalg.cond measures sensitivity; a huge value means unreliable results.
  • np.linalg.lstsq(A, b, rcond=None) returns the least-squares solution, the squared residuals, the rank and the singular values.
  • Fit models by building a design matrix; use np.polyfit and np.polyval for polynomials (highest power first).
  • Always inspect residuals and prefer the simplest model that fits.

Next, look inside matrices themselves with Eigenvalues, SVD and Decompositions.