Reinforcement Learning

Reinforcement learning studies agents that learn by acting in an environment and receiving reward. The useful mental model is: an MDP defines the interaction, value functions evaluate future return, Bellman equations impose consistency, and algorithms differ in how they estimate or optimize those quantities.

Core sequence:

1. Markov Decision Process - the formal environment model: states, actions, transitions, rewards, and discounting.
2. Value Functions - expected return from states or state-action pairs.
3. Bellman Equations - recursive consistency equations for values.
4. Dynamic Programming in RL - exact planning when the transition model is known.
5. Monte Carlo Methods in RL - estimates values from complete sampled returns.
6. Temporal Difference Learning - bootstraps value estimates from partial returns.
7. SARSA - on-policy TD control.
8. Q-Learning - off-policy TD control.
9. Deep Q-Network - uses neural networks to approximate Q-values.
10. Policy Gradient Theorem - differentiates expected return with respect to policy parameters.
11. REINFORCE - Monte Carlo policy gradient.
12. Advantage Function - measures action value relative to state value.
13. Actor-Critic Methods - combines a policy actor with a value critic.
14. Proximal Policy Optimization (PPO) - constrains policy updates for stable actor-critic training.
15. LLMs as RL Agents - applies RL concepts to language-model policies and tool use.

How the pieces fit:

• MDPs define the problem.
• Returns and value functions define what good behavior means.
• Bellman equations connect immediate reward to long-term value.
• Dynamic programming solves known MDPs exactly but usually does not scale to unknown environments.
• Monte Carlo and TD methods estimate values from samples.
• SARSA, Q-learning, and DQN learn action values for control.
• Policy gradients, REINFORCE, actor-critic, and PPO optimize policies directly.
• LLM agent work reuses RL abstractions, but the environment, reward, and credit assignment can be harder to specify cleanly.

Core equations to keep active:

• MDP tuple: $(S,A,P,R,\gamma)$
• Return: $G_t = \sum_{k=0}^{\infty}\gamma^k R_{t+k+1}$
• State value: $V^\pi(s) = \mathbb{E}_\pi[G_t|S_t=s]$
• Action value: $Q^\pi(s,a) = \mathbb{E}_\pi[G_t|S_t=s,A_t=a]$
• Bellman expectation: $V^\pi(s) = \sum_a \pi(a|s)\sum_{s'}P(s'|s,a)[R(s,a,s')+\gamma V^\pi(s')]$
• TD error: $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$
• Q-learning: $Q(s,a) \leftarrow Q(s,a) + \alpha[r+\gamma\max_{a'}Q(s',a')-Q(s,a)]$
• Policy gradient: $\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}[\nabla_\theta \log \pi_\theta(a|s) Q^{\pi_\theta}(s,a)]$
• Advantage: $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$
• PPO ratio: $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t|s_t)}$

See also: Probability, Calculus and Optimization, Neural Networks