RLHF Pipeline

RLHF (Reinforcement Learning from Human Feedback) aligns LLMs with human preferences by training against a learned reward model.

Three-stage pipeline:

Stage 1: SFT

Fine-tune pretrained model on demonstrations of desired behavior

Stage 2: Reward Model Training

Collect comparison data: human ranks multiple model responses to the same prompt
Train a reward model $R_\phi(x, y)$ to predict human preferences using Bradley-Terry model:

P(y_1 \succ y_2) = \sigma(R_\phi(x, y_1) - R_\phi(x, y_2))

The reward model learns what humans consider "good". Its loss $\mathcal{L}(\phi) = -\log P(y_w \succ y_l)$ is exactly binary cross-entropy with the true label fixed to 1 — an LM with a scalar regression head replacing the unembedding

Stage 3: RL Optimization

Optimize the LLM policy $\pi_\theta$ against the reward model using PPO:

\max_\theta \mathbb{E}_{x,y \sim \pi_\theta}[R_\phi(x,y)] - \beta D_{\text{KL}}(\pi_\theta \| \pi_\text{ref})

The KL Penalty $\beta D_\text{KL}$ prevents the model from diverging too far from the SFT model (in practice computed per-token and folded into the per-token reward)

GRPO (critic-free variant): instead of training a value network as the baseline, sample $G$ responses per prompt and use the group statistics. The group-relative advantage for response $i$ is

A^{(i)} = \frac{r^{(i)} - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}

shared across all tokens in that response, then optimized with the same PPO-style clipped objective. This trades extra inference compute ( $G$ completions per prompt) for not having to store/train a critic — used by DeepSeek R1. (Dr. GRPO drops the std and length normalization, which otherwise bias toward easy/hard prompts and over-long wrong answers.)

Why RLHF works:

Easier for humans to compare than to demonstrate (ranking vs. writing)
Captures subtle preferences that supervised data misses
Enables the model to generalize preferences to new situations

Limitations: expensive human annotation, reward model can be gamed (reward hacking), complex pipeline → motivates DPO.

RLHF Pipeline

Linked from