KL Divergence

KL divergence measures how one probability distribution $Q$ diverges from a reference distribution $P$ :

D_{\text{KL}}(P \| Q) = \sum_x P(x) \log\frac{P(x)}{Q(x)} = \mathbb{E}_{x \sim P}\left[\log\frac{P(x)}{Q(x)}\right]

Key properties:

$D_{\text{KL}}(P \| Q) \geq 0$ (Gibbs' inequality), with equality iff $P = Q$
Not symmetric: $D_{\text{KL}}(P \| Q) \neq D_{\text{KL}}(Q \| P)$ — so it's not a true distance
Forward KL $D_{\text{KL}}(P \| Q)$ : $Q$ must cover all of $P$ 's support → mean-seeking
Reverse KL $D_{\text{KL}}(Q \| P)$ : $Q$ avoids regions where $P$ is small → mode-seeking

Where it appears:

Cross-entropy: $H(P, Q) = H(P) + D_{\text{KL}}(P \| Q)$ . Minimizing cross-entropy = minimizing KL from true distribution
RLHF Pipeline — KL penalty $D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ prevents the policy from drifting too far from the reference model → KL Penalty
VAEs — KL term regularizes the latent distribution toward the prior
Information theory — KL is the extra bits needed when using code optimized for $Q$ but the true distribution is $P$

Linked from