Advantage Function

The advantage function measures how much better an action is compared to the average action in that state:

• $A > 0$: action $a$ is better than average under $\pi$
• $A < 0$: action $a$ is worse than average
• $\mathbb{E}_{a \sim \pi}[A^\pi(s,a)] = 0$ by definition

Why use advantage instead of Q:

• Policy gradients with $Q$ or return $G$ have high variance
• $V(s)$ acts as a baseline → using $A = Q - V$ centers the signal, reducing variance without adding bias
• The policy should only increase probability of actions that are better than average, not just positive-return

Generalized Advantage Estimation (GAE):

where $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$ is the Miscoral Difference Learning .

• $\lambda = 0$: pure TD advantage (low variance, some bias)
• $\lambda = 1$: Monte Carlo advantage (no bias, high variance)
• Typical $\lambda = 0.95$ — good bias-variance tradeoff
• GAE is the default advantage estimator in Proximal Policy Optimization (PPO)

See also: Value Functions , Policy Gradient Theorem , Actor-Critic Methods