Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 37 ยท Getting Started

Agents, Environments, States and Rewards

Learn the reinforcement learning vocabulary: state, observation, action, reward, policy, episode, trajectory and return, and build a reset/step environment in Python.

Reinforcement learning has a small vocabulary that you will use in every later lesson. Learn these words once and the rest of the course reads smoothly. We introduce each term, then build a real environment in Python so the words stop being abstract.

The terms

  • Environment: the world the agent lives in. It owns the rules. The agent cannot read its source code; it can only act and observe.
  • State s: a complete description of the situation. In chess, the whole board plus whose turn it is.
  • Observation: what the agent actually sees. It may be less than the state (a poker player cannot see the opponent's cards). When the observation hides information the problem is partially observable. Until chapter 6 we assume the agent sees the full state.
  • Action a: a choice the agent makes. The action space can be discrete (up, down, left, right) or continuous (a steering angle between -30 and +30 degrees).
  • Reward r: a number the environment returns after each step. Positive rewards encourage, negative rewards (costs or penalties) discourage.
  • Policy pi: the agent's behavior, a rule that maps states to actions. A deterministic policy returns one action per state, a stochastic policy returns probabilities.
  • Episode: one complete run from a start state to a terminal state, such as one game. Tasks that end are episodic. Tasks that never end (thermostat control) are continuing.
  • Trajectory: the sequence s0, a0, r1, s1, a1, r2, s2, ... produced by acting. Also called a rollout.
  • Return G: the total reward of a trajectory, possibly discounted. This is what the agent really wants to maximize.

One step of interaction

State s What the world looks like now
Action a The agent picks a move
Reward r A number comes back
Next state The world has changed
1 / 2

What an environment looks like in code

Almost every RL library, including Gymnasium, uses two methods:

  • reset() starts a new episode and returns the first state.
  • step(action) applies the action and returns the next state, the reward, and whether the episode has ended.

We will use this exact shape for our own environments, so code you write here transfers to real libraries. The environment below is a 4x4 grid. The agent starts at the top-left cell, the goal is at the bottom-right, two cells are pits with a penalty, and every other step costs 1 so that short paths are better.

Python A grid environment with reset() and step()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the printout as one trajectory: five steps of reward -1 and a final +10 for reaching the goal. The environment hides how it decided those numbers. A learning agent only ever sees the (state, reward, done) triples.

Notice the done flag. It marks a terminal state: there is nothing left to learn from after it, so algorithms treat the value of the next state as zero.

Policies

A policy decides what to do in each state. Writing one is just writing a function. Here are a deterministic policy, a stochastic one, and a trick every RL program uses: sampling from a probability table.

Python Deterministic and stochastic policies, and the return

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The hand-made policy scores -5 + 10 = 5 in six steps. A random policy averages about -18 (nearly all of its episodes score below -5), because it keeps falling into a pit or wanders for many steps paying 1 each time. The last line previews discounting, covered in returns and discounting: with gamma < 1 a reward that comes later is worth less than one that comes now.

State, observation and what the agent knows

A common beginner mistake is giving the agent too little information. If the state does not contain what is needed to decide, no learning algorithm can recover it. For a robot that must know its speed, one camera frame is not enough, because a still image has no velocity. Fixes include stacking the last few frames, or using memory. The reverse mistake is cluttering the state with irrelevant numbers, which makes learning slow.

When the current state alone is enough to predict what happens next, the state has the Markov property. The next lesson sets up the formal model, in Markov decision processes.

Model-free and model-based

  • A model is the agent's own copy of the environment's rules: given (s, a) it predicts the next state and reward.
  • Model-based methods use or learn a model to plan ahead (dynamic programming, chapter 3).
  • Model-free methods learn values or policies directly from experience, without ever knowing the rules (Monte Carlo, TD learning, Q-learning).

Another split you will see repeatedly: on-policy methods learn about the policy they are currently following, off-policy methods learn about one policy while following another.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does step(action) return in a Gym-style environment?

    step applies the action and reports what happened: the next state, the reward, and whether the episode ended.
  2. 2. What is a policy?

    A policy is the agent's behavior: for each state it gives an action, or a probability of each action.
  3. 3. A task that runs forever, like keeping a room at a set temperature, is called:

    Continuing tasks have no terminal state. Episodic tasks end, for example when a game is over.
  4. 4. A method that learns directly from experience, without knowing the environment's rules, is:

    Model-free methods such as Monte Carlo and Q-learning never use or learn transition probabilities explicitly.

Summary

  • The agent observes a state, picks an action, and the environment returns a reward and the next state.
  • A policy maps states to actions; an episode is one start-to-terminal run; the return is its total reward.
  • Environments are written as reset() and step(action); the done flag marks terminal states.
  • A good state contains everything needed to decide; the Markov property formalizes this.
  • Model-based methods plan with a model of the world, model-free methods learn from experience alone.

Next: Exploration versus Exploitation: Epsilon-Greedy Bandits