Self-improvement methods allow LLMs to improve their own capabilities by generating and learning from their own outputs.
STaR (Self-Taught Reasoner, Zelikman et al., 2022):
- Model attempts to solve problems with chain-of-thought
- Keep solutions that arrive at correct answers
- Fine-tune the model on its own correct reasoning traces
- Repeat — the model bootstraps better reasoning over iterations
Iterative DPO:
- Generate response pairs from the current model
- Score/rank them (with a reward model or verifier)
- Train with DPO on the self-generated preferences
- Repeat with the improved model
When self-improvement works:
- There's a reliable verifier (math, code — can check correctness)
- The model can occasionally produce correct solutions (it has the latent capability)
- Each iteration filters for quality and amplifies good reasoning patterns
When it collapses:
- Without a reliable verifier, the model reinforces its own errors (mode collapse)
- The model may converge to narrow strategies that exploit the verifier
- Diversity decreases across iterations if not managed
- Reward hacking: if the verifier is imperfect, the model learns to fool it
DeepSeek-R1: large-scale demonstration of self-improvement — RL on self-generated reasoning traces, bootstrapped from a small seed of human demonstrations.
See also: Chain-of-Thought Prompting, Test-Time Compute, Direct Preference Optimization