Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 37 ยท Getting Started

What Is Reinforcement Learning?

Learn what reinforcement learning is, how an agent learns from rewards by trial and error, how it differs from supervised learning, and run your first agent.

Reinforcement learning (RL) is the branch of machine learning in which a program learns what to do by doing it. Nobody hands the program a table of correct answers. Instead it tries actions, watches what happens, receives a number called a reward, and gradually works out which behavior produces the most reward over time.

Think of teaching a dog a trick. You cannot explain the trick in words and you cannot show the dog a labeled data set. You wait for something close to what you want, give a treat, and repeat. The dog slowly links its own behavior to treats. RL is that loop written as mathematics and code, and it powers game-playing programs, robot controllers, recommendation systems, data-center cooling, and the final training stage of many large language models.

This lesson gives you the big picture. By the end you will know the parts of an RL problem, how RL differs from the other two kinds of machine learning, and you will have run a tiny agent that really learns.

The core loop

Every RL problem has the same two actors. The agent is the learner and decision maker. The environment is everything the agent cannot directly control. They talk in a loop:

  1. The agent looks at the current state of the environment.
  2. It picks an action.
  3. The environment responds with a new state and a reward.
  4. Repeat.
flowchart LR
    A["Agent<br/>(chooses actions)"] -->|"action"| E["Environment<br/>(changes, pays reward)"]
    E -->|"new state + reward"| A

The agent-environment loop: the agent never sees the rules, only the states and rewards that come back.

The goal is not to maximize the very next reward. It is to maximize the total reward collected over time. That small difference is what makes RL hard and interesting: an action that pays nothing now (opening a door) may be the only way to reach a big payoff later (the treasure room).

Three kinds of machine learning

Supervised learning

Labeled examples Each input comes with the right answer
Goal Predict the label for new inputs
Example Is this photo a cat?
1 / 3

RL differs from supervised learning in four ways that matter in practice:

  • No correct answers. The reward says "that was worth 3", never "the right move was left".
  • Your actions change your data. A supervised model sees a fixed data set. An RL agent decides which situations it experiences, so a bad policy collects bad data.
  • Delayed consequences. A chess move is judged by the game result, many moves later. Working out which earlier decision deserves the credit is the credit assignment problem.
  • Exploration is required. The agent must sometimes try things that look worse to find out whether they really are. You will study this trade-off in exploration versus exploitation.

Your first learning agent

Here is the smallest interesting environment: a corridor of five cells. The agent starts in the middle cell 2 and can step left or right. Reaching cell 4 pays +1 and ends the episode. Reaching cell 0 pays 0 and also ends it. Every other step costs nothing.

A random agent will wander. A learning agent keeps a number for each (cell, action) pair that estimates how good the action is, and nudges that estimate after every step. This is the idea behind Q-learning, which you will meet properly in chapter 5 after the theory is in place. For now just watch it work.

Python A corridor and a learning agent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random agent reaches the goal about half of the time (it is a fair coin walk between two exits), while the learned agent reaches it every time. Nobody told it "go right". It discovered that from the reward alone, and the Q values for the right action grow larger than for the left action in cells 1, 2 and 3.

Where RL is used

  • Games: Backgammon, Go, chess, Atari, StarCraft. Self-play lets an agent generate unlimited experience.
  • Robotics and control: walking, grasping, drone flight, chip floorplanning, cooling of buildings.
  • Operations: inventory control, ad bidding, routing, scheduling.
  • Recommendation and personalization: choosing what to show next when the reward is a click or a purchase.
  • Language models: after pre-training, models are tuned with feedback signals (for example human preferences) using RL algorithms such as policy-gradient methods. Chapter 7 covers the underlying algorithms.

What makes RL hard

  • Sparse rewards. In a long game you may get a single +1 at the very end.
  • Sample inefficiency. Many methods need millions of steps.
  • Instability. The data distribution moves while you learn from it.
  • Reward design. Agents optimize the number you give them, not the goal in your head. A cleaning robot paid per piece of dirt collected may learn to make dirt.

The reward-design problem is easy to see in a toy. Below, a cleaning robot can do two things each hour: clean a dirty tile (+1 reward, 1 tile less dirt) or spill (it makes 2 new dirty tiles). We score two different reward rules and ask which action sequence gets the most total reward in 10 hours.

Python Reward design: the agent optimizes the number you give it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Paying per cleaned tile makes spilling profitable: the best plan spills twice, creating extra dirt to clean, and scores 7 where an honest robot that only cleans the 3 existing tiles scores 3. Paying for a clean floor at the end removes the loophole. The lesson: write rewards that describe the outcome you want, not a proxy that can be gamed.

Throughout this course every example is written in plain Python and NumPy so it runs right here in your browser. We build our own small environments instead of importing a library, which also makes it obvious what an environment really is. Chapter 8 introduces the Gymnasium interface and shows its API with small stand-ins, because the Gymnasium package itself is not available in the in-browser runner.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an RL agent receive after each action?

    The environment answers with the next state and a reward. It never reveals the correct action.
  2. 2. Why is credit assignment hard in RL?

    When rewards are delayed, the agent must work out which of many earlier decisions deserve the credit.
  3. 3. What is the agent trying to maximize?

    RL optimizes long-run return, which is why an action that pays nothing now can still be the best one.
  4. 4. Which statement separates RL from supervised learning?

    A supervised data set is fixed in advance. An RL agent chooses which situations it experiences.

Summary

  • RL is learning to act through trial and error, guided by a reward signal.
  • The agent observes a state, takes an action, and receives a new state and a reward, in a loop.
  • The goal is to maximize total future reward, so delayed consequences and credit assignment matter.
  • Unlike supervised learning there are no correct answers, and the agent's behavior shapes its own data.
  • Exploration, sparse rewards and reward design are the main practical difficulties.

Next: Agents, Environments, States and Rewards