Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 37 ยท Q-learning and SARSA

SARSA: On-policy TD Control

Understand SARSA, the on-policy cousin of Q-learning. Compare both on the cliff-walking problem and see why one learns the risky optimal path and the other the safe one.

Q-learning learns the value of the best policy while behaving exploratively. SARSA makes a different choice: it learns the value of the policy it is actually following, exploration included. The two algorithms differ by a single term in one line of code, yet that term changes the behavior you get, sometimes dramatically.

The name tells you the data it uses: State, Action, Reward, next State, next Action. SARSA needs the action it will really take in the next state before it can update.

The SARSA update

TEXT
Q(s, a)  <-  Q(s, a) + alpha * ( r + gamma * Q(s', a')  -  Q(s, a) )

Compare it with Q-learning, which uses max_a' Q(s', a'). In SARSA, a' is the action chosen by the behavior policy (for example epsilon-greedy) in state s'. The same a' is then executed on the next step.

flowchart LR
    S["s"] --> A["a (chosen by policy)"]
    A --> R["r, s'"]
    R --> A2["a' (chosen by the SAME policy)"]
    A2 --> U["update Q(s,a) with Q(s',a')"]

SARSA is on-policy: the policy that picks a' is the policy being evaluated and improved.

Because the target includes the chance that epsilon-greedy picks a random action next, SARSA's values are pessimistic about risky places. It knows that its own exploration will sometimes make it stumble.

The cliff-walking problem

A classic test: a grid with a cliff along the bottom edge. Stepping into the cliff gives -100 and sends you back to the start. Every other step costs -1.

TEXT
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
S C C C C C C C C C C G      C = cliff

The shortest path hugs the cliff edge. It is optimal when you act greedily, but dangerous while you are still exploring with epsilon > 0, because a random move at the edge means a plunge.

Python SARSA versus Q-learning on the cliff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the two policies and the average online return (the return while still exploring). Typically:

  • Q-learning walks right along the cliff edge. That is the shortest route, so its greedy policy is optimal. But during training it falls off now and then, so its online return is lower.
  • SARSA takes a longer path one or more rows away from the edge. It is not shortest, but it is the best given that the agent keeps exploring, so its online return is higher.

Neither is wrong. They answer different questions: "what is the best possible policy?" versus "what is the best policy for an agent that keeps exploring?"

When to prefer which

Situation Better fit
Learning in a simulator where crashes are free Q-learning (learn the optimum)
Mistakes during learning are costly (a real robot, real money) SARSA (it accounts for its own exploration)
Learning from logged data or someone else's behavior Q-learning (off-policy)
You plan to lower epsilon to zero over time Both converge to the same optimum if exploration decays

If you decay epsilon toward zero (called GLIE: greedy in the limit with infinite exploration), SARSA also converges to the optimal policy.

Seeing the difference in the numbers

You can quantify the risk directly. Train both agents, then evaluate each one greedily (no exploration) and with 10% random actions:

Python Greedy versus exploring evaluation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Greedy evaluation favors Q-learning (the shortest path), while exploratory evaluation favors SARSA (the safe path). Same environment, two honest answers.

Common mistakes

  • Choosing a' for the update, then choosing a different action when you actually step. In SARSA the same a' must be executed.
  • Forgetting that the final transition should not bootstrap.
  • Comparing the algorithms by their online return only: that metric rewards caution, not optimality.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the only difference between the SARSA and Q-learning targets?

    SARSA bootstraps from the next action the policy really picks, Q-learning from the best next action.
  2. 2. Why is SARSA called on-policy?

    The policy that selects actions (epsilon-greedy) is the policy whose values SARSA estimates.
  3. 3. In cliff walking with epsilon = 0.1, which path does a converged SARSA agent usually prefer?

    Its values include the cost of occasional random moves, so risky cells next to the cliff look worse.
  4. 4. What condition makes SARSA converge to the optimal policy?

    With Greedy in the Limit with Infinite Exploration, the behavior policy becomes the greedy one, so SARSA's target becomes the optimal one.

Summary

  • SARSA updates with r + gamma * Q(s', a'), where a' is the action it will really take.
  • It is on-policy: it learns the value of its own exploratory behavior.
  • On the cliff, Q-learning finds the shortest risky path, SARSA a safer longer one.
  • With decaying exploration both reach the optimal policy.

Next: Exploration Strategies