Skip to content
Tutorials Center
My learning

Topics

Stochastic Processes Tutorial

Learn Stochastic Processes step by step: 36 lessons in 9 chapters with examples, quizzes, diagrams and practice.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is a Stochastic Process? Learn what a stochastic process is: a collection of random variables indexed by time. See coin flips, queues and prices as processes, with runnable Python.
  2. 2 States, Time and Sample Paths Understand state spaces, index sets, sample paths, finite-dimensional distributions and information over time, with Python plots of random-walk paths.
  3. 3 Simulating Stochastic Processes in Python Learn to simulate stochastic processes with NumPy: seeds, vectorized paths, a reusable simulator and Monte Carlo estimates checked against exact answers.
  4. 4 Classifying Stochastic Processes A map of stochastic processes: discrete or continuous time and state, Markov, stationary, independent increments and martingales, with a Python check.

Random Variables Review

  1. 5 Probability and Expectation Review Review random variables, distributions, expectation, variance and independence, plus the six distributions you need for stochastic processes, with Python checks.
  2. 6 Conditional Probability and Conditional Expectation Master conditioning: Bayes rule, the law of total probability, the law of total expectation and random sums, with Python simulations that confirm each formula.
  3. 7 Generating Functions Learn probability generating functions: definition, moments from derivatives, sums of independent variables and random sums, verified symbolically with SymPy.
  4. 8 Law of Large Numbers and the Central Limit Theorem See why averages settle down and sums look normal: Markov and Chebyshev bounds, the law of large numbers and the central limit theorem, tested in Python.

Markov Chains

  1. 9 The Markov Property and Transition Matrices Learn what a Markov chain is: the Markov property, states, transition probabilities and the transition matrix, with a weather chain you simulate in Python.
  2. 10 n-Step Transitions and the Chapman-Kolmogorov Equations Compute multi-step probabilities of a Markov chain with matrix powers: the Chapman-Kolmogorov equations and the distribution of X_n, checked in Python.
  3. 11 Classification of States: Communicating Classes, Recurrent and Transient Classify Markov chain states: accessibility, communicating classes, closed and absorbing classes, recurrent versus transient, with a Python class finder.
  4. 12 Absorbing Chains and Hitting Probabilities Analyze absorbing Markov chains: canonical form, the fundamental matrix N = (I - Q)^-1, expected time to absorption and absorption probabilities, in Python.
  5. 13 First-Step Analysis and Gambler's Ruin Solve expected hitting times and ruin probabilities with first-step analysis, derive the gambler's ruin formulas and verify them with linear algebra and simulation.

Stationary Distributions and Mixing

  1. 14 Stationary Distributions Find the stationary distribution of a Markov chain: the equation pi P = pi, three ways to solve it in Python, what it means and when it exists or is unique.
  2. 15 Irreducibility, Periodicity and Ergodicity Learn why some chains never settle: the period of a state, aperiodic and ergodic chains, and a Python function that computes periods and tests ergodicity.
  3. 16 The Convergence Theorem and the Ergodic Theorem Why ergodic chains forget their start: the convergence theorem, the ergodic theorem for time averages and the mean return time 1/pi, tested in Python.
  4. 17 Mixing Time and Total Variation Distance Measure how fast a Markov chain forgets its start: total variation distance, mixing time, the second eigenvalue and the cycle walk example, computed in Python.
  5. 18 Reversibility and Detailed Balance Learn detailed balance and reversible Markov chains: a shortcut for stationary distributions, birth-death chains and the Metropolis algorithm, checked in Python.

Poisson Processes

  1. 19 The Poisson Process: Counting Random Events in Time Learn what a Poisson process is, why counts in a time window follow a Poisson distribution, and check the idea with a binomial limit and a simulation.
  2. 20 Interarrival and Arrival Times See why waiting times between Poisson events are exponential, why the n-th arrival is Erlang, and how memorylessness works, all checked by simulation.
  3. 21 Superposition and Thinning of Poisson Processes Merge independent Poisson streams, split one stream by coin flips, and use competing exponentials. Both rules are proved by intuition and checked in code.
  4. 22 Nonhomogeneous and Compound Poisson Processes Handle rates that change over time with the thinning algorithm, and add random sizes to events with a compound Poisson process. Includes runnable simulations.

Continuous-Time Markov Chains

  1. 23 The Generator Matrix of a Continuous-Time Markov Chain Learn continuous-time Markov chains through the generator matrix Q: rates, zero row sums, and the matrix exponential P(t) = exp(Qt), with runnable code.
  2. 24 Holding Times and the Jump Chain A CTMC stays in each state for an exponential time, then jumps by a discrete chain. Simulate it, and find the long-run fraction of time by solving ฯ€Q = 0.
  3. 25 Birth-Death Processes Study birth-death chains where the state moves up or down by one. Derive the product-form stationary distribution and test it on queues and a pure-birth process.
  4. 26 Kolmogorov Equations and Transient Behavior Derive the forward and backward Kolmogorov equations, solve them numerically with Euler steps and uniformization, and watch a chain approach equilibrium.

Random Walks and Brownian Motion

  1. 27 The Simple Random Walk Meet the simple random walk: its mean and variance, how far it wanders, the exact probability of returning to zero, and why dimension decides recurrence.
  2. 28 Gambler's Ruin A gambler plays until broke or at a target. Derive ruin probabilities and expected game length for fair and unfair games, and check them by simulation.
  3. 29 The Reflection Principle and the Maximum of a Walk Use the reflection principle to count random walk paths that touch a level. Derive P(max โ‰ฅ a) for the symmetric walk and verify it with exact counting.
  4. 30 Brownian Motion Basics Get Brownian motion as the scaling limit of a random walk. Learn its defining properties, covariance min(s,t), quadratic variation, and simulate it.

Applications and Simulation

  1. 31 Simulating Continuous-Time Chains with Gillespie's Method Write the Gillespie algorithm from scratch, simulate an immigration-death chain, and verify it against exact stationary and transient results.
  2. 32 Simulating the M/M/1 Queue Analyze the M/M/1 queue with Poisson arrivals and exponential service, derive L, W and Little's law, and verify them with a short Lindley recursion simulation.
  3. 33 Branching Processes and Extinction Model a growing population with the Galton-Watson process, find the mean generation size, and compute the extinction probability as a fixed point.

Projects and Practice

  1. 34 Practice Set: Poisson Processes, CTMCs and Random Walks Eight worked practice problems on Poisson processes, continuous-time chains, random walks and Brownian motion, each solved with runnable code you can check.
  2. 35 Project: A Bank Queue Simulator Build a simulator for a bank with several tellers, compare it with the exact Erlang C formula, and use it to decide how many tellers you need.
  3. 36 Project: Stock Price Paths with Geometric Brownian Motion Simulate stock-like price paths with geometric Brownian motion, check their mean and spread, and price a European call by Monte Carlo against the Black-Scholes formula.