Proximal Policy Optimization (PPO)

PPO is the default policy gradient algorithm for practical RL, including RLHF.

The problem PPO solves: standard policy gradients can make destructively large updates that ruin the policy. Trust region methods (TRPO) constrain the update size but are complex.

PPO-Clip objective:

where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_\text{old}}(a_t|s_t)}$ is the probability ratio and $\hat{A}_t$ is the GAE advantage.

The ratio $r_t$ comes from off-policy reuse: rollouts are sampled once from $\pi_{\theta_\text{old}}$ and reused for several gradient steps, with $r_t$ acting as an importance-sampling correction. Clipping keeps the approximation valid as long as $\pi_\theta$ hasn't moved too far from $\pi_{\theta_\text{old}}$.

How clipping works:

• If $\hat{A}_t > 0$ (good action): $r_t$ is clipped at $1+\epsilon$ → limits how much we increase the probability
• If $\hat{A}_t < 0$ (bad action): $r_t$ is clipped at $1-\epsilon$ → limits how much we decrease the probability
• Typical $\epsilon = 0.2$

The four cases (clipping is asymmetric — it only stops you when already moving the way the advantage points):

• $r_t > 1+\epsilon,\ A_t > 0$ → use $(1+\epsilon)A_t$, gradient 0: stop pushing an already-boosted good action further
• $r_t < 1-\epsilon,\ A_t < 0$ → use $(1-\epsilon)A_t$, gradient 0: stop pushing an already-suppressed bad action down
• $r_t > 1+\epsilon,\ A_t < 0$ → use $r_t A_t$, gradient flows: still allowed to correct a bad action
• $r_t < 1-\epsilon,\ A_t > 0$ → use $r_t A_t$, gradient flows: still allowed to recover a good action

Why PPO is the default:

• Simple to implement (just a clipped objective, no constraint optimization)
• Stable — prevents catastrophic policy collapses
• Good sample efficiency with multiple epochs over the same batch
• Works across diverse domains (games, robotics, LLM alignment)

In RLHF: PPO optimizes the LLM policy against a learned reward model, with a KL Penalty to prevent reward hacking.

See also: Actor-Critic Methods, Policy Gradient Theorem, RLHF Pipeline