# topic/ai-ml/rl

Actor-Critic Methods

Actor-critic combines a policy (actor) with a value function (critic) for stable, low-variance policy gradient learning.

Advantage Function

The advantage function measures how much better an action is compared to the average action in that state:

Bellman Equations

The Bellman equations express the recursive relationship in Value Functions: the value of a state equals the immediate reward plus the discounted value of the next state.

Deep Q-Network

DQN extends Q-Learning to high-dimensional state spaces by approximating with a neural network.

Dynamic Programming in RL

Dynamic programming solves MDPs exactly when the model ( , ) is known. Foundation for understanding all RL algorithms.

LLMs as RL Agents

The framing of LLM generation as a reinforcement learning problem reveals deep connections between language modeling and sequential decision-making.

Markov Decision Process

An MDP is the formal framework for sequential decision-making under uncertainty.

Monte Carlo Methods in RL

Monte Carlo (MC) methods learn value functions from complete episodes of experience — no model needed.

Policy Gradient Theorem

The policy gradient theorem gives the gradient of the expected return with respect to policy parameters — enabling direct optimization of the policy.

Proximal Policy Optimization (PPO)

PPO is the default policy gradient algorithm for practical RL, including RLHF.

Q-Learning

Q-learning is an off-policy TD control algorithm that directly learns the optimal action-value function .

REINFORCE

REINFORCE is the simplest policy gradient algorithm. It uses complete episode returns to estimate the gradient.

Reinforcement Learning

Reinforcement learning studies agents that learn by acting in an environment and receiving reward. The useful mental model is: an MDP defines the interaction, value functions…

RLHF Pipeline

RLHF (Reinforcement Learning from Human Feedback) aligns LLMs with human preferences by training against a learned reward model.

SARSA

SARSA is an on-policy TD control algorithm. The name comes from the tuple .

Temporal Difference Learning

TD learning combines the model-free nature of Monte Carlo with the bootstrapping of dynamic programming.

Value Functions

Value functions estimate how good it is to be in a state (or to take an action in a state) under a given policy’