Value Functions

Value functions estimate how good it is to be in a state (or to take an action in a state) under a given policy’

State-value function $V^\pi(s)$:

"Expected return starting from state $s$ and following policy $\pi$."

Action-value function $Q^\pi(s,a)$:

"Expected return starting from state $s$, taking action $a$, then following $\pi$."

Relationship: $V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s,a)$ — the value of a state is the expected Q-value over actions.

Optimal value functions:

• $V^*(s) = \max_\pi V^\pi(s)$ — best possible value
• $Q^*(s,a) = \max_\pi Q^\pi(s,a)$ — best possible Q-value
• Optimal policy: $\pi^*(a|s) = 1$ if $a = \arg\max_a Q^*(s,a)$ — just be greedy w.r.t. $Q^*$

Key insight: if you know $Q^’$ , the optimal policy is trivial — just take the action with highest $Q$-value. This is why Q-Learning focuses on learning $Q^’$.

See also: Bellman Equations , Markov Decision Process , Advantage Function