Stochastic Processes Tutorial
Learn Stochastic Processes step by step: 36 lessons in 9 chapters with examples, quizzes, diagrams and practice.
Start learning →
of lessons completed
Getting Started
- What Is a Stochastic Process? Learn what a stochastic process is: a collection of random variables indexed by time. See coin flips, queues and prices as processes, with runnable Python.
- States, Time and Sample Paths Understand state spaces, index sets, sample paths, finite-dimensional distributions and information over time, with Python plots of random-walk paths.
- Simulating Stochastic Processes in Python Learn to simulate stochastic processes with NumPy: seeds, vectorized paths, a reusable simulator and Monte Carlo estimates checked against exact answers.
- Classifying Stochastic Processes A map of stochastic processes: discrete or continuous time and state, Markov, stationary, independent increments and martingales, with a Python check.
Random Variables Review
- Probability and Expectation Review Review random variables, distributions, expectation, variance and independence, plus the six distributions you need for stochastic processes, with Python checks.
- Conditional Probability and Conditional Expectation Master conditioning: Bayes rule, the law of total probability, the law of total expectation and random sums, with Python simulations that confirm each formula.
- Generating Functions Learn probability generating functions: definition, moments from derivatives, sums of independent variables and random sums, verified symbolically with SymPy.
- Law of Large Numbers and the Central Limit Theorem See why averages settle down and sums look normal: Markov and Chebyshev bounds, the law of large numbers and the central limit theorem, tested in Python.
Markov Chains
- The Markov Property and Transition Matrices Learn what a Markov chain is: the Markov property, states, transition probabilities and the transition matrix, with a weather chain you simulate in Python.
- n-Step Transitions and the Chapman-Kolmogorov Equations Compute multi-step probabilities of a Markov chain with matrix powers: the Chapman-Kolmogorov equations and the distribution of X_n, checked in Python.
- Classification of States: Communicating Classes, Recurrent and Transient Classify Markov chain states: accessibility, communicating classes, closed and absorbing classes, recurrent versus transient, with a Python class finder.
- Absorbing Chains and Hitting Probabilities Analyze absorbing Markov chains: canonical form, the fundamental matrix N = (I - Q)^-1, expected time to absorption and absorption probabilities, in Python.
- First-Step Analysis and Gambler's Ruin Solve expected hitting times and ruin probabilities with first-step analysis, derive the gambler's ruin formulas and verify them with linear algebra and simulation.
Stationary Distributions and Mixing
- Stationary Distributions Find the stationary distribution of a Markov chain: the equation pi P = pi, three ways to solve it in Python, what it means and when it exists or is unique.
- Irreducibility, Periodicity and Ergodicity Learn why some chains never settle: the period of a state, aperiodic and ergodic chains, and a Python function that computes periods and tests ergodicity.
- The Convergence Theorem and the Ergodic Theorem Why ergodic chains forget their start: the convergence theorem, the ergodic theorem for time averages and the mean return time 1/pi, tested in Python.
- Mixing Time and Total Variation Distance Measure how fast a Markov chain forgets its start: total variation distance, mixing time, the second eigenvalue and the cycle walk example, computed in Python.
- Reversibility and Detailed Balance Learn detailed balance and reversible Markov chains: a shortcut for stationary distributions, birth-death chains and the Metropolis algorithm, checked in Python.
Poisson Processes
- The Poisson Process: Counting Random Events in Time Learn what a Poisson process is, why counts in a time window follow a Poisson distribution, and check the idea with a binomial limit and a simulation.
- Interarrival and Arrival Times See why waiting times between Poisson events are exponential, why the n-th arrival is Erlang, and how memorylessness works, all checked by simulation.
- Superposition and Thinning of Poisson Processes Merge independent Poisson streams, split one stream by coin flips, and use competing exponentials. Both rules are proved by intuition and checked in code.
- Nonhomogeneous and Compound Poisson Processes Handle rates that change over time with the thinning algorithm, and add random sizes to events with a compound Poisson process. Includes runnable simulations.
Continuous-Time Markov Chains
- The Generator Matrix of a Continuous-Time Markov Chain Learn continuous-time Markov chains through the generator matrix Q: rates, zero row sums, and the matrix exponential P(t) = exp(Qt), with runnable code.
- Holding Times and the Jump Chain A CTMC stays in each state for an exponential time, then jumps by a discrete chain. Simulate it, and find the long-run fraction of time by solving ฯQ = 0.
- Birth-Death Processes Study birth-death chains where the state moves up or down by one. Derive the product-form stationary distribution and test it on queues and a pure-birth process.
- Kolmogorov Equations and Transient Behavior Derive the forward and backward Kolmogorov equations, solve them numerically with Euler steps and uniformization, and watch a chain approach equilibrium.
Random Walks and Brownian Motion
- The Simple Random Walk Meet the simple random walk: its mean and variance, how far it wanders, the exact probability of returning to zero, and why dimension decides recurrence.
- Gambler's Ruin A gambler plays until broke or at a target. Derive ruin probabilities and expected game length for fair and unfair games, and check them by simulation.
- The Reflection Principle and the Maximum of a Walk Use the reflection principle to count random walk paths that touch a level. Derive P(max โฅ a) for the symmetric walk and verify it with exact counting.
- Brownian Motion Basics Get Brownian motion as the scaling limit of a random walk. Learn its defining properties, covariance min(s,t), quadratic variation, and simulate it.
Applications and Simulation
- Simulating Continuous-Time Chains with Gillespie's Method Write the Gillespie algorithm from scratch, simulate an immigration-death chain, and verify it against exact stationary and transient results.
- Simulating the M/M/1 Queue Analyze the M/M/1 queue with Poisson arrivals and exponential service, derive L, W and Little's law, and verify them with a short Lindley recursion simulation.
- Branching Processes and Extinction Model a growing population with the Galton-Watson process, find the mean generation size, and compute the extinction probability as a fixed point.
Projects and Practice
- Practice Set: Poisson Processes, CTMCs and Random Walks Eight worked practice problems on Poisson processes, continuous-time chains, random walks and Brownian motion, each solved with runnable code you can check.
- Project: A Bank Queue Simulator Build a simulator for a bank with several tellers, compare it with the exact Erlang C formula, and use it to decide how many tellers you need.
- Project: Stock Price Paths with Geometric Brownian Motion Simulate stock-like price paths with geometric Brownian motion, check their mean and spread, and price a European call by Monte Carlo against the Black-Scholes formula.