Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Environments, Gymnasium and Applications

Gymnasium: The Standard RL Environment API

Learn the Gymnasium API used by almost all RL code: reset and step, spaces, wrappers, the terminated versus truncated distinction and how to write and check your own environment.

Every RL algorithm needs an environment to interact with, and nearly every library and paper uses the same interface: Gymnasium (the maintained successor of OpenAI Gym, now run by the Farama Foundation). Learn this small API once and you can plug any agent into hundreds of ready-made environments, or into your own.

This lesson runs the real Gymnasium 1.x in your browser. Gymnasium imports a multiprocessing module that browsers do not provide, so every example starts with a short stand-in for _multiprocessing. You will not need it on your own computer: there, pip install gymnasium is all it takes.

The environment loop

An environment has two core methods:

  • obs, info = env.reset(seed=...) starts a new episode and returns the first observation.
  • obs, reward, terminated, truncated, info = env.step(action) applies an action and returns what happened.

The agent-environment loop from the first lessons then looks like this:

Python
obs, info = env.reset(seed=0)
done = False
while not done:
    action = policy(obs)
    obs, reward, terminated, truncated, info = env.step(action)
    done = terminated or truncated

Terminated versus truncated

The two flags mean different things, and mixing them up is one of the most common bugs in RL code:

  • terminated is True when the episode ended because of the environment's own rules (the pole fell, the goal was reached). The next state truly has value zero.
  • truncated is True when it ended for an outside reason, usually a time limit. The agent was not in a terminal state, so the value of the next state is not zero and you must still bootstrap from it.
Python A random agent on CartPole

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A random agent keeps the pole up for only about 20 steps. A good one reaches the 500-step limit, at which point truncated becomes True.

Spaces

Spaces describe what observations and actions look like:

Space Meaning Example
Discrete(n) integers 0 .. n-1 CartPole actions: push left or right (Discrete(2))
Box(low, high, shape) real-valued arrays CartPole observation, a vector of four floats
MultiDiscrete([...]) several discrete choices a gamepad with several buttons
MultiBinary(n) n on/off flags keyboard keys held
Dict, Tuple, Text combinations structured observations

Useful methods: space.sample() draws a random element and space.contains(x) checks validity. The shape of the spaces tells you which agent design fits: a Discrete action space suits Q-learning or a softmax policy, a Box action space needs a Gaussian policy (see the policy gradient lesson).

Writing your own environment

To make an environment, subclass gym.Env, declare observation_space and action_space, and implement reset and step. Here is a small corridor: the agent starts at a random cell and must reach the right end, but each step costs a little. The code ends by running Gymnasium's own check_env, which verifies the API contract, and by training a tabular Q-learning agent on it.

Python A custom Gymnasium environment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The agent learns to go right in every cell (the last cell is the goal itself, so its action is never used). Notice the line bootstrap = 0.0 if terminated else .... We test terminated, not done, because a truncated episode did not really end and its next state still has value.

Wrappers

A wrapper is an environment that wraps another one and modifies it, without changing the original code. Gymnasium ships many:

  • TimeLimit(env, max_episode_steps): truncate after a number of steps.
  • RecordEpisodeStatistics: adds episode return and length to info.
  • NormalizeObservation, NormalizeReward: running normalization.
  • FrameStackObservation: stack the last k frames (a common trick for pixel inputs).
  • ClipAction, RescaleAction: make continuous actions fit the allowed range.

Write your own by subclassing gym.RewardWrapper, gym.ObservationWrapper or gym.ActionWrapper. The main idea is composition: env = Wrapper2(Wrapper1(base_env)).

Vectorized environments

Deep RL algorithms such as A2C and PPO run many copies of the environment in parallel with gym.make_vec("CartPole-v1", num_envs=8); all observations and rewards arrive as arrays of shape (num_envs, ...), and finished sub-environments reset automatically. (Parallel worker processes are not available in the browser, so run those on your own machine.)

Rendering and the built-in catalog

Pass render_mode="human" (a window) or "rgb_array" (frames as arrays, for video) to gym.make. The built-in environments are grouped into Classic Control (CartPole, MountainCar, Pendulum, Acrobot), Toy Text (FrozenLake, Taxi, CliffWalking, Blackjack), Box2D (LunarLander, CarRacing), MuJoCo (physics-based robots like Hopper and Ant) and Atari. Box2D, MuJoCo and Atari need extra packages that are not available in this browser sandbox.

Common mistakes

  • Using done = terminated or truncated as the bootstrapping condition. Cut the bootstrap only on terminated.
  • Forgetting super().reset(seed=seed) in a custom reset, so seeding does nothing.
  • Returning an observation that is not inside observation_space (wrong dtype or shape). check_env catches this.
  • Mutating the observation array you returned; return a copy.
  • Using the old Gym 4-tuple (obs, reward, done, info) API. Gymnasium returns five values.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does env.step(action) return in Gymnasium?

    Gymnasium splits the old 'done' flag into terminated and truncated.
  2. 2. An episode ends because of a 200-step time limit. What is the correct handling when computing a TD target?

    Truncation is not a property of the task; cutting the bootstrap there biases the values downward.
  3. 3. Which space type describes the four real numbers observed in CartPole?

    Box holds real-valued arrays with bounds; Discrete holds integers.
  4. 4. What is a wrapper?

    Wrappers compose, so you can add a time limit and normalization without editing the base environment.

Summary

  • Gymnasium's reset() and step() are the common interface for agents and environments.
  • terminated marks a real terminal state; truncated marks an external cut-off, and only the former zeros the bootstrap.
  • Spaces (Discrete, Box, ...) describe observations and actions and guide algorithm choice.
  • A custom environment needs two spaces plus reset and step; check_env validates it.
  • Wrappers and vectorized environments add features and speed without touching the base environment.

Next: Reward Design and Shaping