Test-time compute refers to strategies that spend more computation at inference to improve output quality, rather than scaling the model itself.
Methods:
Best-of-N sampling: generate responses, score with a reward/verifier model, return the best. Simple but effective — linear in compute, often logarithmic in quality improvement.
Self-consistency: generate multiple chain-of-thought paths, take the majority-vote answer. Reduces variance of any single reasoning chain.
Tree search: explore a tree of reasoning steps, using a process reward model to guide which branches to expand. Related to MCTS (Monte Carlo Tree Search) from game-playing RL.
Iterative refinement: generate an answer, critique it, revise. Repeat for rounds.
Why test-time compute matters:
- Training compute has diminishing returns (fixed model, Scaling Laws)
- Test-time compute can be allocated adaptively — hard problems get more compute
- DeepSeek-R1: demonstrated that extended "thinking" (long CoT) dramatically improves reasoning
- A small model with lots of test-time compute can match a larger model on reasoning tasks
The tradeoff: test-time compute is more expensive per query but more flexible. You can scale it per-problem rather than per-model.
Connection to RL: test-time search is analogous to planning in RL — look ahead before committing to an action.
See also: Chain-of-Thought Prompting, Self-Improvement in LLMs, Scaling Laws