Deep Q-Network

DQN extends Q-Learning to high-dimensional state spaces by approximating $Q(s,a)$ with a neural network.

Core idea: $Q(s,a;\theta) \approx Q^*(s,a)$ — a neural network takes state $s$ as input and outputs Q-values for all actions.

Two critical innovations (Mnih et al., 2015):

1. Experience Replay:

• Store transitions $(s, a, r, s')$ in a replay buffer
• Sample random mini-batches for training
• Breaks temporal correlations in sequential data → more stable, sample-efficient
• Each experience can be reused multiple times

2. Target Network:

• Maintain a separate target network $Q(s,a;\theta^-)$ that is periodically copied from the main network
• TD target: $y = r + \gamma \max_{a'} Q(s', a'; \theta^-)$
• Prevents the moving target problem (training toward a constantly changing target)
• Updated every $C$ steps: $\theta^- \leftarrow \theta$

Loss: MSE between predicted Q-value and TD target:

Variants:

• Double DQN — decouple action selection (main network) from evaluation (target network) to reduce overestimation
• Dueling DQN — separate value and advantage streams: $Q(s,a) = V(s) + A(s,a)$
• Prioritized replay — sample transitions with high TD error more frequently

See also: Q-Learning, REINFORCE, Advantage Function