Direct Preference Optimization

DPO (Rafailov et al., 2023) trains directly on preference pairs without a separate reward model or RL loop.

Key insight: the KL-regularized RLHF objective has a closed-form optimum,

\pi^*(y|x) = \frac{1}{Z(x)}\pi_\text{ref}(y|x)\exp\!\left(\tfrac{1}{\beta}r(x,y)\right)

Rearranging to solve for the reward gives

r^*(x, y) = \beta \log\frac{\pi^*(y|x)}{\pi_\text{ref}(y|x)} + \beta\log Z(x)

The intractable partition function $Z(x)$ is the same for $y_w$ and $y_l$ , so it cancels in the Bradley-Terry preference probability — letting us reparameterize the reward in terms of the policy itself and drop the reward model entirely.

DPO loss:

L_\text{DPO}(\theta) = -\mathbb{E}_{(x,y_w,y_l)}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_\text{ref}(y_l|x)}\right)\right]

where $y_w$ = preferred response, $y_l$ = dispreferred response.

Intuition: increase the relative probability of preferred responses over dispreferred ones, relative to the reference model.

DPO vs RLHF:

	RLHF	DPO
Reward model	Required (separate training)	Implicit
RL loop	PPO (complex, unstable)	Supervised loss (simple)
Stability	Harder to tune	More stable
Performance	Potentially more powerful	Competitive, sometimes better

DPO is simpler: one supervised training stage on preference data, no separate reward model, no RL. This has made it the default for many alignment setups.

Direct Preference Optimization

Linked from