Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 68 ยท Linear Algebra and Numerical Methods

Matrices and Linear Systems

Matrix operations, Gaussian elimination, determinants, inverses and numpy.linalg.solve, plus why ill-conditioned systems give unreliable answers.

Almost every engineering model eventually becomes a pile of linear equations: currents in a circuit network, forces in a truss, node temperatures in a heat solver, the normal equations of a curve fit. Write them together and you get a single matrix equation A x = b. The finite-difference heat solver from the last lesson was secretly a matrix multiplication at every step; implicit versions require solving A x = b at every step.

This lesson covers the working skills: what a matrix does, how elimination solves a system, what the determinant and inverse tell you, how to call numpy.linalg.solve, and why some systems cannot be trusted even though a computer returns an answer.

Matrices as machines

A matrix A with m rows and n columns is a table of numbers that turns an n-vector into an m-vector: y = A x. Each entry of y is a row of A dotted with x.

Key operations:

  • Add and scale entry by entry (same shapes only).
  • Multiply A B: entry (i, j) is row i of A dotted with column j of B. The inner sizes must match (m x n times n x p gives m x p).
  • Transpose A.T swaps rows and columns.
  • Identity I has ones on the diagonal; A I = I A = A.

Matrix multiplication is not commutative: A B and B A usually differ, and one may not even exist. It is associative, so (A B) C = A (B C). Geometrically, a matrix stretches, rotates, shears or projects space, and multiplying matrices chains those actions.

Python Matrix basics in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the order reversal in (A B).T = B.T A.T, a rule that appears constantly in derivations.

Linear systems and Gaussian elimination

Take the system

Text
 2x +  y -  z =  8
-3x -  y + 2z = -11
-2x +  y + 2z = -3

In matrix form A x = b with A the table of coefficients. Gaussian elimination solves it by row operations that never change the solution:

  1. Swap two rows.
  2. Multiply a row by a nonzero number.
  3. Add a multiple of one row to another.

Use the first row to clear the entries below the first pivot, then the second row to clear below the second pivot, until A becomes upper triangular. Then back-substitution reads off the unknowns from the bottom row upward. For n equations the cost is about n**3 / 3 operations, trivial for small systems and the reason very large systems need smarter methods.

flowchart LR
    A["A x = b"] --> B["forward elimination<br/>(clear below pivots)"]
    B --> C["triangular system<br/>U x = c"]
    C --> D["back-substitution"]
    D --> E["solution x"]

Elimination turns a tangled system into one you can solve from the last equation upward.

In practice we swap rows so the pivot (the entry we divide by) is the largest available. This partial pivoting keeps round-off small and handles zero pivots.

Python Gaussian elimination from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The solution is x = 2, y = 3, z = -1. The second system has the second row equal to twice the first: one equation is redundant, so there is no unique answer, and the code reports a singular matrix.

Determinants

For a square matrix, the determinant det(A) is a single number that answers: does A squash space? For a 2x2 matrix [[a, b], [c, d]] it is a*d - b*c. Geometrically |det A| is the factor by which A scales areas (2D) or volumes (3D), and a negative sign means the orientation flips.

The vital fact: A x = b has exactly one solution for every b if and only if det(A) != 0. If det(A) = 0, the matrix collapses space onto a lower dimension (a line, or a plane), the columns are linearly dependent, and the system has either no solutions or infinitely many. Such a matrix is singular.

Other useful rules: det(A B) = det(A) det(B), det(A.T) = det(A), and swapping two rows flips the sign. In practice, nobody computes a determinant by the cofactor formula for anything beyond 3x3; elimination does it in n**3 steps (it is the product of the pivots, up to sign).

Inverses and solving systems

If det(A) != 0, there is an inverse A^-1 with A^-1 A = A A^-1 = I, and the solution is x = A^-1 b. For 2x2:

inverse of [[a, b], [c, d]] = 1/(a*d - b*c) * [[d, -b], [-c, a]]

Do not compute inverses to solve systems. np.linalg.solve(A, b) does elimination directly, which is faster and more accurate than forming np.linalg.inv(A) @ b. Compute an inverse only when you truly need the inverse matrix itself, such as in a covariance formula.

Python Determinant, inverse and solve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The determinant of A is 4*3 - 3*6 = -6: areas scale by 6 and orientation flips. The solution is x = 1, y = 2.

Ill-conditioning: when an answer is not an answer

A computer cannot store most numbers exactly (see Floating-Point Error and Conditioning). Usually that is harmless, but some systems amplify tiny input errors enormously. A matrix is ill-conditioned when it is nearly singular: its columns are almost parallel, so small changes in b or A swing the solution wildly.

The condition number cond(A) measures this. Roughly, relative error in the solution can be up to cond(A) times the relative error in the data. With 16 digits of double precision, a condition number of 10**10 leaves only about 6 trustworthy digits. A tiny determinant is not the right test (scaling a matrix changes its determinant but not its conditioning); use np.linalg.cond.

Python A nearly singular system

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Changing b in the fifth significant digit moves the solution from (1, 1) to (0, 2), a change as large as the answer itself. The well-conditioned matrix barely reacts. Hilbert matrices grow so ill-conditioned that by size 12 the condition number exceeds 10**16, meaning double precision gives essentially no correct digits.

What to do when conditioning is poor: rescale variables so columns have similar size, reformulate the problem (for least squares, use QR or SVD rather than normal equations), collect better-spread data, or accept that the question is not well posed and say so.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Matrix A is 3x2 and matrix B is 2x4. What is the shape of A @ B?

    Inner sizes (2 and 2) match, and the result has the outer sizes: 3 rows by 4 columns.
  2. 2. What does det(A) = 0 tell you about the system A x = b?

    A zero determinant means A is singular. Its columns are dependent, so it cannot be inverted.
  3. 3. Why is np.linalg.solve(A, b) preferred over np.linalg.inv(A) @ b?

    solve uses factorization with pivoting directly. Forming the inverse costs more and adds round-off error.
  4. 4. Which number best warns that a linear system is unreliable?

    The condition number measures how much the solution can amplify errors in the data. The determinant alone is misleading because it changes under scaling.
  5. 5. In Gaussian elimination, why is partial pivoting used?

    Choosing the largest available pivot keeps multipliers small, which keeps round-off under control, and it swaps away zero pivots.

Summary

  • A matrix maps vectors to vectors; products chain these maps and are not commutative.
  • Gaussian elimination reduces A x = b to a triangular system solved by back-substitution, at a cost of about n**3 / 3 operations; partial pivoting makes it reliable.
  • det(A) = 0 means singular: no unique solution. |det A| is the area or volume scaling factor.
  • Use np.linalg.solve, not the inverse, to solve systems.
  • A large condition number (np.linalg.cond) signals that small data errors become large solution errors.

Next: Eigenvalues and Eigenvectors in Engineering.