SARSA

2 min read

# topic/ai-ml/rl

SARSA is an on-policy TD control algorithm. The name comes from the tuple (St,At,Rt+1,St+1,At+1)(S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1}).

Update rule:

Q(St,At)←Q(St,At)+α[Rt+1+γQ(St+1,At+1)−Q(St,At)]Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha\left[R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t)\right]

On-policy: unlike Q-Learning's max⁡a′\max_{a'}, SARSA uses the action At+1A_{t+1} that the agent actually takes. It learns the value of the policy it's following (including exploration).

SARSA vs Q-Learning:

SARSAQ-Learning
TargetR+γQ(S′,A′)R + \gamma Q(S', A') (actual next action)R+γmax⁡a′Q(S′,a′)R + \gamma \max_{a'} Q(S', a') (best action)
PolicyOn-policyOff-policy
BehaviorLearns to be safe while exploringLearns optimal regardless of exploration

Classic example (CliffWalking):

  • Q-learning learns the optimal path (right along the cliff edge) — risky during exploration
  • SARSA learns a safer path (far from the cliff) — accounts for its own ϵ\epsilon-greedy exploration

When to use SARSA: when the evaluation policy and behavior policy should be the same, or when you care about safety during learning (the learned policy reflects exploration risks).

Expected SARSA: uses ∑aπ(a∣s′)Q(s′,a)\sum_a \pi(a|s')Q(s',a) instead of Q(s′,A′)Q(s', A') — reduces variance, interpolates between SARSA and Q-learning.

See also: Q-Learning, Miscoral Difference Learning, Actor-Critic Methods

Linked from