LLMs as RL Agents

The framing of LLM generation as a reinforcement learning problem reveals deep connections between language modeling and sequential decision-making.

The MDP mapping:

• State = current context (prompt + generated tokens so far)
• Action = next token (from vocabulary)
• Policy = the language model $\pi_\theta(a_t|s_t)$
• Reward = task success, human preference score, or reward model output
• Episode = one complete generation

Implications:

• Pretraining = learning a prior policy from demonstrations (behavioral cloning on internet text)
• SFT = imitation learning on expert demonstrations
• RLHF = online RL with a learned reward function
• DPO = offline RL from preference data
• Token-level generation is an MDP with vocabulary-sized action space

This framing enables:

• Chain-of-Thought Prompting = reasoning as sequential actions
• ReAct = LLM as an agent that takes actions in an environment
• Process reward models = reward shaping at each reasoning step (not just the final answer)
• Self-Improvement in LLMs = self-play and iterative improvement

Key insight: all the RL concepts (value functions, policy gradients, exploration-exploitation, reward shaping) apply directly to language models. The token-level view unifies language modeling and RL.

See also: Markov Decision Process, RLHF Pipeline, Chain-of-Thought Prompting