Actor-Critic Methods

Actor-critic combines a policy (actor) with a value function (critic) for stable, low-variance policy gradient learning.

Two networks:

• Actor $\pi_\theta(a|s)$ — the policy. Updated via Policy Gradient Theorem
• Critic $V_\phi(s)$ or $Q_\phi(s,a)$ — evaluates the actor's actions. Updated via Miscoral Difference Learning

Update loop:

1. Actor takes action $a \sim \pi_\theta(\cdot|s)$, observes $r, s'$
2. Critic computes TD error: $\delta = r + \gamma V_\phi(s') - V_\phi(s)$
3. Update critic: $\phi \leftarrow \phi + \alpha_c \delta \nabla_\phi V_\phi(s)$
4. Update actor: $\theta \leftarrow \theta + \alpha_a \delta \nabla_\theta \log\pi_\theta(a|s)$

The TD error $\delta$ serves as an estimate of the Advantage Function .

Why actor-critic is better than pure policy gradient:

• REINFORCE needs full episodes and has high variance
• The critic provides a low-variance baseline (bootstrapping via TD)
• Can update at every step (online), not just at episode end

Variants:

• A2C (Advantage Actor-Critic): synchronous, uses Advantage Function instead of raw TD error
• A3C: asynchronous parallel actors for faster training
• Proximal Policy Optimization (PPO) : clips the policy ratio to prevent destructive updates — the default for RLHF

See also: REINFORCE , Advantage Function , Proximal Policy Optimization (PPO)