Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Environments, Gymnasium and Applications

Where RL Is Used: Applications, Safety and Beyond

A tour of where reinforcement learning works in practice and what lies beyond the basics: model-based RL, multi-agent learning, offline RL, RLHF, sim-to-real, safety and limits, with two runnable demos.

You now know the core algorithms: tabular and deep value methods, policy gradients, actor-critic and PPO. This lesson looks outward. Where does RL actually deliver value? What other branches of the field exist? And what should you keep in mind before using it on a real problem, where mistakes cost money or safety?

Where RL has worked

Area Example Why RL fits
Games Backgammon, Atari, Go, chess, StarCraft, poker Clear rewards, a perfect simulator, unlimited self-play
Robotics Locomotion, manipulation, drone racing Continuous control with a physical simulator to train in
Data centers and chips Cooling control, chip floorplanning A simulator or a safe, slowly changing system
Recommendation and ads Sequential recommendation, bidding Long-term engagement rather than a single click
Operations Inventory, routing, scheduling Sequential decisions under uncertainty
Language models RLHF, reasoning models trained against verifiable rewards The reward comes from human preferences or a checker
Science and engineering Plasma control, protocol design, molecule design Expensive experiments that can be simulated

Notice the common thread: a good simulator or a lot of safe, cheap interaction. RL needs millions of trials, so where trials are expensive or dangerous (medicine, finance, real roads), people lean on the more careful variants below.

Model-based RL

All of the deep methods so far were model-free. Model-based RL learns a model of the environment (transitions and rewards) and uses it for planning, as in Dyna-Q. The attraction is sample efficiency: a good model lets the agent practice in its head. The risk is model error: the planner exploits mistakes in the model and produces plans that fail in reality. Modern systems such as MuZero and Dreamer learn compact latent models and plan within them.

The simplest model-based method is certainty equivalence: estimate the model from data, then solve the estimated MDP exactly as if the estimate were true. Let us see how the quality of the result depends on how much data was used. The environment is a 5-by-5 grid with random slipping (20 percent of moves go sideways); we collect transitions using a random policy, estimate the model by counting, run value iteration on it, and score the resulting policy in the true environment.

Python Certainty equivalence: how much data does a model need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With little data the planned policy can be poor, and with more it approaches the optimum, without any trial-and-error learning of values. The model was learned from random data, which is a form of off-policy learning, one of the big advantages of model-based approaches.

Multi-agent RL

When several agents learn at the same time, each one's environment includes the others, and since they are all changing, the environment is non-stationary from every agent's point of view. Settings range from fully cooperative (a team of robots) to fully competitive (two-player zero-sum games) to mixed (self-driving cars sharing a road). Approaches include self-play (agent trains against copies of itself), centralized training with decentralized execution, and communication between agents. Game-theoretic ideas such as Nash equilibrium replace the single-agent notion of optimality.

A tiny demonstration of what makes it different: two agents play rock-paper-scissors using regret matching, a self-play method that picks each move with probability proportional to its positive regret.

Python Self-play on rock-paper-scissors with regret matching

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The average strategies approach the equilibrium, even though the agents' current strategies keep chasing each other. That is a general theme: in competitive games, the average over time, not the latest policy, is what converges.

Offline RL

Often you cannot interact at all: you only have a logged dataset (clinical records, past ad impressions, robot logs). Offline (batch) RL learns a policy from that fixed data. The trap is distribution shift: the learned policy may choose actions that the data barely contains, and the value estimates for those actions are pure extrapolation, and usually too optimistic (this is maximization bias again). Methods such as Conservative Q-Learning (CQL) and behavior-regularized methods penalize or avoid actions outside the data. Offline policy evaluation estimates how good a new policy would be using only logs, via importance sampling (see importance sampling for off-policy learning).

Imitation learning and learning from human feedback

  • Behavior cloning treats expert demonstrations as supervised data. Simple, but errors compound because the learner visits states the expert never did.
  • DAgger fixes that by letting the learner act and asking the expert for corrections in the states it reaches.
  • Inverse RL infers the reward function that explains expert behavior.
  • RLHF trains a reward model from human comparisons ("which answer is better?") and then optimizes a policy (a language model) against it with PPO, with a KL penalty to keep the model close to its original behavior. Newer variants skip the explicit reward model (direct preference optimization) or use automatic checkers (verifiable rewards) as the reward.

Sim-to-real, safety and responsibility

  • Sim-to-real transfer: train in simulation, deploy in the real world. Domain randomization randomizes physics, textures and delays in simulation so the policy copes with the real gap.
  • Safe RL adds constraints (do not leave this region, keep the cost below a budget) through constrained optimization, shielding or conservative exploration.
  • Reward misspecification (see reward design) is a safety problem as well as a performance problem.
  • Reproducibility: results vary greatly between random seeds and implementation details. Report several seeds with confidence intervals, tune baselines as hard as your method, and evaluate with the exploration off.
  • Ethics: systems that optimize engagement or profit can cause harm that the reward does not capture. Include humans in the loop whenever decisions affect people.

When not to use RL

RL is powerful but expensive in data, time and tuning. Before reaching for it, ask:

  1. Is it really a sequential decision problem where actions affect future states? If not, a supervised model or a bandit may be enough.
  2. Is there a simple rule or an optimization method (dynamic programming, control theory, linear programming) that solves it exactly?
  3. Do you have a simulator or a safe way to collect lots of interaction?
  4. Can you write down a reward that really expresses the goal?

If the answers are mostly "no", RL will probably frustrate you.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main advantage of model-based RL?

    A learned model lets the agent practice offline, though planning with a wrong model can exploit its errors.
  2. 2. Why is multi-agent learning harder than single-agent learning?

    From one agent's view, the other learners are part of the environment, and they keep changing.
  3. 3. What is the central risk in offline RL?

    Without interaction, errors on out-of-distribution actions cannot be corrected, so methods stay conservative.
  4. 4. What is the role of the reward model in RLHF?

    Humans compare outputs; a model trained on those comparisons supplies the reward that PPO then optimizes.

Summary

  • RL shines where there is a good simulator or cheap safe interaction: games, robotics, control, operations, and tuning language models.
  • Model-based methods trade sample efficiency against model error; multi-agent settings are non-stationary and use ideas from game theory.
  • Offline RL, imitation learning and RLHF replace or supplement online trial and error.
  • Safety, reward design, reproducibility and ethics are part of the engineering, not an afterthought.
  • Before using RL, check whether a simpler method solves the problem.

Next: Project: Tic-Tac-Toe by Self-Play