Bellman Equations

1 min read

# topic/ai-ml/rl

The Bellman equations express the recursive relationship in Value Functions: the value of a state equals the immediate reward plus the discounted value of the next state.

Bellman expectation equation (for policy π\pi):

Vπ(s)=∑aπ(a∣s)[R(s,a)+γ∑s′P(s′∣s,a)Vπ(s′)]V^\pi(s) = \sum_a \pi(a|s) \left[R(s,a) + \gamma \sum_{s'} P(s'|s,a) V^\pi(s')\right] Qπ(s,a)=R(s,a)+γ∑s′P(s′∣s,a)∑a′π(a′∣s′)Qπ(s′,a′)Q^\pi(s,a) = R(s,a) + \gamma \sum_{s'} P(s'|s,a) \sum_{a'} \pi(a'|s') Q^\pi(s',a')

Bellman optimality equation (for optimal policy):

V∗(s)=max⁡a[R(s,a)+γ∑s′P(s′∣s,a)V∗(s′)]V^*(s) = \max_a \left[R(s,a) + \gamma \sum_{s'} P(s'|s,a) V^*(s')\right] Q∗(s,a)=R(s,a)+γ∑s′P(s′∣s,a)max⁡a′Q∗(s′,a′)Q^*(s,a) = R(s,a) + \gamma \sum_{s'} P(s'|s,a) \max_{a'} Q^*(s',a')

Why this matters:

  • The recursive structure enables Dynamic Programming in RL: solve by iterating the Bellman equation
  • Q-Learning uses a sample-based approximation of the Bellman optimality equation
  • Miscoral Difference Learning bootstraps by using current value estimates in place of true expected values
  • The Bellman equation is to RL what the chain rule is to deep learning — the fundamental recursive decomposition

Fixed point: V∗V^* is the unique fixed point of the Bellman optimality operator (contraction mapping theorem guarantees convergence).

See also: Value Functions, Dynamic Programming in RL, Q-Learning

Linked from