Linear Algebra Tutorial
Learn Linear Algebra step by step: 62 lessons in 10 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is Linear Algebra? Find out what linear algebra is, where it shows up in graphics, machine learning, physics and search, and how this track uses NumPy to make every idea runnable.
- Vectors Learn what vectors are, how to read them as arrows, points and lists, and how to add, subtract and scale them by hand and with NumPy arrays.
- Length, Distance and Unit Vectors Measure vectors with the L1, L2 and max norms, compute Euclidean distance, normalize to unit vectors and use np.linalg.norm correctly with examples.
- The Dot Product Understand the dot product as a sum of products and as a measure of alignment, then find angles, cosine similarity, projections and orthogonal vectors in NumPy.
- The Cross Product Learn the 3D cross product, the right-hand rule, how its length gives parallelogram area, how to find normal vectors, and how to use np.cross safely.
Matrices
- Matrices Learn what a matrix is, how rows, columns and shapes work, and how to build identity, diagonal, triangular and symmetric matrices with NumPy.
- Matrix Addition and Scalar Multiplication Learn the simple matrix operations: adding matrices, scaling them, taking the transpose and trace, and the shape rules that decide which operations are allowed.
- Matrix Multiplication Learn matrix multiplication: the row-by-column rule, shape compatibility, why AB is not BA, the difference between @ and *, associativity and the cost of products.
- Three Ways to See Matrix-Vector Products See the product Ax three ways: as row dot products, as a combination of the columns of A, and as a linear map that transforms vectors, with NumPy examples.
- Transpose and Symmetric Matrices Learn the rules of the transpose, why (AB)^T reverses the order, how to recognize symmetric and skew-symmetric matrices, and why A^T A is always symmetric.
- Special Matrices: Skew-Symmetric, Orthogonal, Idempotent, Nilpotent and More A field guide to special matrices: definitions, how to recognize each type, and the determinant and eigenvalue facts that follow, with a NumPy classifier.
- Block Matrices and Partitioned Products Split big matrices into blocks, multiply and invert them block by block, and use the Schur complement to get determinants and inverses, checked in NumPy.
Linear Systems
- Systems of Linear Equations Understand systems of linear equations as lines meeting and as column combinations, write them as Ax = b, and learn when there are one, no or infinitely many solutions.
- Gaussian Elimination Learn Gaussian elimination: the three row operations, echelon form, back substitution and partial pivoting, then write a working solver from scratch in Python.
- Reduced Row Echelon Form Learn reduced row echelon form: pivots, free variables, parametric solutions and a short RREF function in NumPy that you can reuse on any system.
- Solving Systems with NumPy Use np.linalg.solve to solve Ax = b, learn why you should not compute the inverse, check results with residuals, and meet conditioning warnings.
- Inverse Matrices Understand what a matrix inverse is, when it exists, how to compute it by hand and with np.linalg.inv, its algebra rules, and why singular matrices fail.
- Cramer's Rule and the Adjugate Learn Cramer's rule and the adjugate (adjoint) formula for the inverse, work through 2x2 and 3x3 examples in Python, and see why elimination beats them in practice.
- LU Decomposition Factor a matrix as A = PLU, solve Ax = b with two cheap triangular solves, reuse the factors for many right-hand sides, and use scipy.linalg.lu_factor.
- Iterative Solvers: Jacobi, Gauss-Seidel and Conjugate Gradient Solve large linear systems without factoring: implement Jacobi, Gauss-Seidel and conjugate gradient, learn the convergence tests and compare speeds in Python.
Vector Spaces
- Vector Spaces and Subspaces Learn the ten vector space axioms, the three-step subspace test, and why polynomials, matrices and functions are vector spaces too, with NumPy checks.
- Linear Combinations and Span Learn linear combinations and the span of vectors, see what they fill in 2D and 3D, and test with NumPy whether a vector b lies in the span of others.
- Linear Independence Learn what linearly independent and dependent vectors are, how to test them with matrix rank in NumPy, and how to find and remove redundant vectors.
- Basis and Dimension Learn what a basis is, how to find coordinates in any basis, how dimension counts the free directions of a space, and how to test bases with NumPy.
- Column Space and Null Space Meet the four fundamental subspaces of a matrix, compute bases for the column space and null space with SciPy, and see how solutions to Ax = b are structured.
- Rank Understand matrix rank as the number of independent directions, use the rank-nullity theorem, and learn why np.linalg.matrix_rank needs a tolerance.
Linear Transformations
- Linear Transformations See a matrix as a function that moves vectors, learn the two rules of linearity, and read a matrix straight from where it sends the basis vectors.
- Rotation, Scaling, Shear and Reflection Build the standard 2D matrices for rotation, scaling, shear and reflection, compose them in the right order, extend to 3D rotations and plot with matplotlib.
- Homogeneous Coordinates and Translation Add one extra coordinate so translation becomes a matrix product, build 3x3 affine matrices for 2D, chain them, and see the idea behind graphics pipelines.
- Change of Basis Learn how coordinates change when you switch bases, how to write a matrix in a new basis with P^-1 A P, and why similar matrices share trace, determinant and eigenvalues.
Determinants and Orthogonality
- Determinants Understand the determinant as an area and volume scaling factor, learn its sign and properties, compute it by hand and with NumPy, and avoid underflow with slogdet.
- Inner Product Spaces, Norms and Cauchy-Schwarz Generalize the dot product to inner products on matrices and functions, learn the norm axioms, and prove and test Cauchy-Schwarz and the triangle inequality.
- Orthogonality and Projections Learn orthogonal vectors and subspaces, project a vector onto a line or a subspace with the normal equations, and verify the projection matrix properties in NumPy.
- Gram-Schmidt and QR Decomposition Build orthonormal bases with the Gram-Schmidt process, compare classical and modified versions for stability, and use np.linalg.qr to factor and solve systems.
- Least Squares Solve overdetermined systems with least squares: normal equations, np.linalg.lstsq, fitting a line and a parabola, reading residuals, and why the error is orthogonal.
Eigenvalues and Decompositions
- Eigenvalues and Eigenvectors Learn what eigenvalues and eigenvectors are, find them with the characteristic polynomial and np.linalg.eig, see the geometry, and verify Av = lambda v in code.
- Gershgorin Circles and Eigenvalue Bounds Use the Gershgorin circle theorem to locate eigenvalues from the rows of a matrix, test invertibility with diagonal dominance, and bound the spectral radius.
- Cayley-Hamilton Theorem Understand the Cayley-Hamilton theorem: every matrix satisfies its characteristic polynomial. Use it for inverses, powers and the minimal polynomial, with Python checks.
- Diagonalization and Matrix Powers Learn diagonalization A = P D P^-1: compute matrix powers fast, get Fibonacci numbers from eigenvalues, and spot defective matrices that cannot be diagonalized.
- Markov Chains and Stochastic Matrices See how stochastic matrices model Markov chains, find the steady state as the eigenvector for eigenvalue 1, measure convergence speed and handle absorbing chains in Python.
- Symmetric Matrices and the Spectral Theorem Why symmetric matrices have real eigenvalues and orthogonal eigenvectors, how to use np.linalg.eigh, and how quadratic forms and positive definiteness follow.
- Cholesky Decomposition Learn the Cholesky factorization A = L L^T for positive definite matrices: the algorithm, a worked example, solving systems, and sampling correlated data.
- The Covariance Matrix Learn the covariance matrix: variances on the diagonal, covariances off it, 2x2 and 3x3 examples by hand, positive semi-definiteness and its link to PCA.
- Complex Matrices: Conjugate Transpose, Hermitian and Unitary Extend linear algebra to complex numbers: conjugate transpose, complex inner products, Hermitian and unitary matrices, the Fourier matrix, all in NumPy.
- Singular Value Decomposition Understand the SVD A = U Sigma V^T: its geometry, np.linalg.svd shapes, how singular values reveal rank and condition number, and the pseudoinverse.
- Polar Decomposition Split any matrix into a rotation (or reflection) and a stretch with the polar decomposition A = UP, computed from the SVD and used for the nearest orthogonal matrix.
- Jordan Normal Form and Schur Decomposition Handle defective matrices with generalized eigenvectors and the Jordan form, then use the always-available Schur decomposition A = U T U*, with SymPy and SciPy examples.
- Low-Rank Approximation and Image Compression Use the truncated SVD to build the best rank-k approximation of a matrix, understand the Eckart-Young idea, and compress and denoise a generated image.
- Numerical Linear Algebra Pitfalls Learn why floating-point linear algebra goes wrong: rounding, ill-conditioned systems, the Hilbert matrix, unstable algorithms, and sparse matrices in SciPy.
- Tensors, Kronecker Products and Einsum Go beyond matrices: understand tensors as multi-dimensional arrays, the Kronecker product and its eigenvalue rules, the vec identity and einsum contractions in NumPy.
Applications
Projects
- Project: Principal Component Analysis Build principal component analysis from scratch with NumPy: covariance matrix, eigenvectors, SVD, projection, explained variance, and a scikit-learn check.
- Project: PageRank Build PageRank from scratch: turn links into a stochastic matrix, run power iteration, add damping and dangling pages, and check the result against an eigenvector.
- Project: A Matrix-Factorization Recommender Build a movie recommender with linear algebra: a sparse ratings matrix, low-rank user and item factors, alternating least squares, held-out error and a rank sweep.
- Project: A Tiny 3D Wireframe Renderer Draw a spinning 3D cube using only matrices: vertex arrays, rotation matrices, a camera shift, perspective projection and matplotlib line drawing.
- Project: Linear Regression from Scratch Fit linear regression four ways with NumPy: normal equations, QR, SVD and lstsq. See why conditioning matters, then tame collinear features with ridge regression.
Practice and Review
- Practice: Vectors and Matrices Twelve linear algebra exercises on vectors, norms, dot and cross products, matrix building, multiplication and transposes. Every solution runs live in NumPy.
- Practice: Systems and Vector Spaces Twelve exercises on linear systems, elimination, inverses, LU, span, independence, basis, null space and rank. Every solution runs live in your browser.
- Practice: Eigenvalues and Decompositions Ten graded exercises on eigenvalues, diagonalization, the spectral theorem, positive definiteness and the SVD, each with a runnable NumPy solution.
- Common Mistakes and Debugging The most common linear algebra bugs in NumPy: shape errors, * versus @, misusing inv, comparing floats with ==, and eigenvector sign ambiguity, with fixes.
- Interview Questions Classic linear algebra interview questions for data science and machine learning roles, with clear answers and runnable NumPy coding solutions.
- Linear Algebra Cheat Sheet A one-page linear algebra cheat sheet with key concepts, formulas and the matching NumPy and SciPy calls for vectors, matrices, systems, eigenvalues and the SVD.