Skip to content
Tutorials Center
My learning

Topics

Reinforcement Learning Tutorial

Learn Reinforcement Learning step by step: 37 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is Reinforcement Learning? Learn what reinforcement learning is, how an agent learns from rewards by trial and error, how it differs from supervised learning, and run your first agent.
  2. 2 Agents, Environments, States and Rewards Learn the reinforcement learning vocabulary: state, observation, action, reward, policy, episode, trajectory and return, and build a reset/step environment in Python.
  3. 3 Exploration vs Exploitation and Epsilon-Greedy Bandits Understand the exploration-exploitation trade-off with the multi-armed bandit problem: sample-average estimates, epsilon-greedy action selection, optimistic starts and regret.
  4. 4 UCB, Thompson Sampling and Softmax Bandits Go beyond epsilon-greedy: implement UCB1, Thompson sampling with Beta distributions and softmax gradient bandits in Python, and compare their regret.

MDPs and Bellman Equations

  1. 5 Markov Decision Processes Learn the MDP, the mathematical model behind reinforcement learning: states, actions, transition probabilities, rewards, the Markov property and how to simulate one in Python.
  2. 6 Returns, Discounting and the Goal of RL Understand the return G_t, the discount factor gamma, episodic versus continuing tasks, the recursive return formula and how gamma shapes the behavior an agent learns.
  3. 7 Policies and Value Functions Learn what a policy is and how the state-value function v and action-value function q measure long-term goodness, with a Python grid world estimated by simulation.
  4. 8 Bellman Expectation Equations Derive the Bellman expectation equations for v and q, see why they are a system of linear equations, and solve them exactly with NumPy for a grid world and a small MDP.
  5. 9 Bellman Optimality Equations and Optimal Policies Learn optimal value functions v* and q*, the Bellman optimality equations, why the max makes them non-linear, and how to extract an optimal policy; verified with Python code.

Dynamic Programming

  1. 10 Iterative Policy Evaluation (Prediction with DP) Compute the value function of a policy by sweeping the Bellman expectation backup: learn expected updates, convergence tests, in-place sweeps and see it work on a grid world.
  2. 11 Policy Improvement and Policy Iteration Learn the policy improvement theorem and policy iteration: alternate evaluation and greedy improvement until the policy is stable, implemented step by step on a slippery grid world.
  3. 12 Value Iteration and the Gambler's Problem Learn value iteration, the Bellman optimality backup as an update rule: truncated policy evaluation, convergence tests, policy extraction and the classic gambler's problem in Python.
  4. 13 Asynchronous DP, Generalized Policy Iteration and Limits of DP Learn asynchronous dynamic programming, prioritized sweeping and generalized policy iteration, compare update orders in Python and see why DP struggles with the curse of dimensionality.

Monte Carlo and Temporal Difference

  1. 14 Monte Carlo Prediction: Learning Values from Episodes Learn Monte Carlo prediction: estimate value functions by averaging sampled returns, first-visit versus every-visit, incremental means, with a grid world and a blackjack simulator in Python.
  2. 15 Monte Carlo Control: Exploring Starts and Epsilon-Soft Policies Find optimal policies with Monte Carlo control: estimate action values q, improve greedily, guarantee exploration with exploring starts or epsilon-soft policies, with blackjack and grid world code.
  3. 16 Off-Policy Learning and Importance Sampling Learn the difference between on-policy and off-policy learning, behavior and target policies, importance sampling ratios, ordinary versus weighted importance sampling and their variance, with Python.
  4. 17 Temporal-Difference Learning: TD(0) Prediction Learn TD(0), the core idea of reinforcement learning: update value estimates after every step by bootstrapping, the TD error, and how TD compares with Monte Carlo on the random walk task in Python.
  5. 18 n-step TD and Eligibility Traces (TD-lambda) Bridge TD(0) and Monte Carlo with n-step returns and TD(lambda): the lambda-return, eligibility traces and the backward view, compared on a 19-state random walk with runnable Python.

Q-learning and SARSA

  1. 19 The Q-learning Algorithm Learn tabular Q-learning: the Q-table, the update rule, off-policy learning and epsilon-greedy exploration, with a complete gridworld agent you can run and edit.
  2. 20 SARSA: On-policy TD Control Understand SARSA, the on-policy cousin of Q-learning. Compare both on the cliff-walking problem and see why one learns the risky optimal path and the other the safe one.
  3. 21 Exploration Strategies Compare the main exploration strategies in reinforcement learning: epsilon-greedy and decay, softmax, UCB, optimistic initialization and count bonuses, with runnable experiments.
  4. 22 Expected SARSA and Double Q-learning Learn two upgrades to tabular TD control: Expected SARSA lowers update variance, and Double Q-learning removes maximization bias. Runnable Python experiments included.
  5. 23 Dyna-Q: Learning and Planning Learn Dyna-Q: combine real experience with a learned model to plan in the background, and see how planning steps speed up learning on a maze with runnable Python code.

Function Approximation and Deep RL

  1. 24 Function Approximation: Why Tables Run Out See why Q-tables fail on big problems and how function approximation generalizes: state aggregation, semi-gradient TD, features and the deadly triad, with runnable Python.
  2. 25 Tile Coding and Mountain Car Build tile coding from scratch and train a linear SARSA agent on the Mountain Car task: sparse features, overlapping tilings, step-size scaling and a runnable agent in Python.
  3. 26 Deep Q-Networks from Scratch Build a Deep Q-Network step by step in NumPy: a small neural network for Q-values, experience replay, a target network and the full DQN training loop on a runnable gridworld.
  4. 27 DQN Improvements: Double, Dueling, Prioritized Replay Learn the main upgrades to Deep Q-Networks: Double DQN, dueling heads, prioritized experience replay, n-step returns, noisy nets and Rainbow, with small runnable NumPy demos.

Policy Gradients and Actor-Critic

  1. 28 The Policy Gradient Theorem and REINFORCE Learn policy gradient methods: parametrized policies, softmax and Gaussian policies, the log-derivative trick, the policy gradient theorem and a runnable REINFORCE agent in NumPy.
  2. 29 Baselines and Variance Reduction Learn why REINFORCE is noisy and how a baseline fixes it: proof that it adds no bias, a variance experiment in NumPy, learned value baselines and advantage normalization.
  3. 30 Actor-Critic and A2C Learn actor-critic methods: a policy (actor) guided by a value function (critic), the TD error as advantage, A2C and A3C, generalized advantage estimation, with runnable NumPy code.
  4. 31 PPO and Trust Regions Learn why large policy updates collapse performance and how TRPO and PPO fix it: the surrogate objective, the clipped ratio, multiple epochs per batch and a runnable NumPy PPO agent.

Environments, Gymnasium and Applications

  1. 32 Gymnasium: The Standard RL Environment API Learn the Gymnasium API used by almost all RL code: reset and step, spaces, wrappers, the terminated versus truncated distinction and how to write and check your own environment.
  2. 33 Reward Design and Shaping Learn how to design rewards: sparse versus dense rewards, reward hacking, potential-based shaping that keeps the optimal policy, and runnable NumPy experiments that show the effect.
  3. 34 Where RL Is Used: Applications, Safety and Beyond A tour of where reinforcement learning works in practice and what lies beyond the basics: model-based RL, multi-agent learning, offline RL, RLHF, sim-to-real, safety and limits, with two runnable demos.

Projects and Practice

  1. 35 Project: Tic-Tac-Toe by Self-Play Build a tic-tac-toe agent that learns purely by playing against itself: afterstate values, temporal-difference learning, exploration, and an evaluation against random play and a perfect minimax opponent.
  2. 36 Project: Balancing CartPole from Scratch Build the CartPole physics yourself, then train policies that balance the pole with random search and the cross-entropy method, using only NumPy and a simple linear policy.
  3. 37 Practice: Debugging RL and Self-Test Learn how to test and debug RL code: sanity tests on tiny MDPs, seed variance and confidence intervals, a debugging checklist, a mistakes catalog and practice exercises with answers.