KL Penalty

The KL penalty in RLHF constrains the trained policy from drifting too far from the reference (SFT) model.

In the RLHF objective:

Why it's essential — without KL penalty:

• The model finds "adversarial" outputs that score high on the reward model but are actually bad
• Reward hacking: the model exploits imperfections in the learned reward model
• Examples: generating very long responses (length bias), using sycophantic language, repeating phrases the reward model likes
• The outputs become degenerate — high reward, low quality

What KL penalty does:

• Keeps $\pi_\theta$ close to $\pi_\text{ref}$ in distribution space
• Acts as a regularizer: "be better, but don't change too much"
• $\beta$ controls the tradeoff: higher $\beta$ = more conservative, lower $\beta$ = more optimization
• In practice, $\beta$ is often adapted during training

Connection to KL Divergence:

• $D_\text{KL}(\pi_\theta \| \pi_\text{ref}) = \mathbb{E}_{y \sim \pi_\theta}\left[\log\frac{\pi_\theta(y|x)}{\pi_\text{ref}(y|x)}\right]$
• Measures how many "bits" of difference between the two policies
• Penalizes deviations proportional to how unlikely the output would be under the reference

See also: RLHF Pipeline, KL Divergence, Proximal Policy Optimization (PPO)