Policy Gradient Theorem

The policy gradient theorem gives the gradient of the expected return with respect to policy parameters — enabling direct optimization of the policy.

Objective: $J(\theta) = \mathbb{E}{\pi\theta}[G_0]$ - maximize expected return

The theorem:

Derivation intuition:

• $\nabla_\theta \log\pi_\theta(a|s)$ = direction to increase probability of action $a$ in state $s$
• $Q^{\pi_\theta}(s,a)$ = how good action $a$ actually was
• The product: increase probability of good actions, decrease probability of bad ones
• The expectation: average over the states and actions the policy visits

Why this is important:

• Works for continuous action spaces (unlike value-based methods that need $\arg\max$)
• Can learn stochastic policies (useful for partially observable environments)
• Directly optimizes the objective (no need for a value function, though one helps → Actor-Critic Methods )

Derivation (log-derivative trick): the key identity is $\nabla_\theta P = P \nabla_\theta \log P$, which lets the gradient pass through the sampling distribution. In trajectory form with $J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]$:

The transition dynamics drop out of $\nabla_\theta \log P(\tau|\theta)$ because they don't depend on $\theta$ — only the policy terms survive.

Baselines: subtracting a baseline $b(s)$ from $Q$: $\nabla_\theta J \propto \mathbb{E}[\nabla\log\pi \cdot (Q(s,a) - b(s))]$ — this doesn't change the expectation but reduces variance. It stays unbiased because $b$ depends only on the state: $\mathbb{E}_{a \sim \pi}[\nabla_\theta \log\pi_\theta(a|s)\,b(s)] = b(s)\nabla_\theta!\sum_a \pi_\theta(a|s) = b(s)\nabla_\theta 1 = 0$. The optimal baseline is approximately $V(s)$, giving the Advantage Function .

See also: REINFORCE , Advantage Function , Proximal Policy Optimization (PPO)