Recurrent Neural Networks

RNNs process sequences by maintaining a hidden state that is updated at each time step:

The same weights $(W_h, W_x)$ are shared across all time steps (parameter sharing through time).

Forward pass: process sequence left to right, updating $\mathbf{h}$ at each step. Output can be taken at each step (sequence-to-sequence) or only at the end (sequence-to-one).

Backpropagation through time (BPTT): unroll the RNN into a deep network (one layer per time step) and apply Backpropagation. The gradient flows through all time steps.

Vanishing gradient problem: the gradient passes through $W_h$ at every time step. With $z_t = W_x\mathbf{x}_t + W_h\mathbf{h}_{t-1} + \mathbf{b}$, the step-to-step Jacobian is

Since $\tanh' \in (0, 1]$ the diagonal factor only ever shrinks, and repeated multiplication by $W_h$ drives gradients toward 0 (if $\|W_h\| < 1$) or $\infty$ (if $\|W_h\| > 1$). So the RNN forgets long-range dependencies; explosion is fixed by gradient clipping.

This is the fundamental limitation of vanilla RNNs — they struggle with sequences longer than ~20-50 steps.

Solutions:

• LSTM and GRU — gating mechanisms to control information flow
• Transformers — direct position-to-position connections, no sequential bottleneck

See also: LSTM and GRU, Self-Attention, Backpropagation