Chain-of-thought (CoT) prompting improves LLM reasoning by eliciting intermediate steps before the final answer.
Key finding (Wei et al., 2022): adding "Let's think step by step" or providing few-shot examples with reasoning steps dramatically improves performance on math, logic, and commonsense reasoning tasks.
Why it works:
- LLMs are autoregressive — each token conditions on all previous tokens
- Intermediate reasoning tokens provide a "scratchpad" that the model can attend to
- Decomposing a hard problem into easier sub-problems lets the model solve each step within its capabilities
- Without CoT, the model must jump directly from question to answer in one "forward pass"
Process Reward Models (PRMs):
- Instead of rewarding only the final answer, grade each reasoning step
- Enables training models to reason more reliably
- Can detect and penalize invalid reasoning steps even if the final answer is correct
- Used in DeepSeek-R1 and other reasoning-focused models
Self-consistency: generate multiple CoT paths, take the majority-vote answer. Reduces errors from any single reasoning chain.
Relation to RL: CoT turns reasoning into a sequential decision process where each step is an action. This connects to LLMs as RL Agents and Test-Time Compute (spending more compute at inference to reason better).
See also: LLMs as RL Agents, Test-Time Compute, ReAct