LSTM and GRU

LSTMs and GRUs solve the vanishing gradient problem of Recurrent Neural Networks through gating mechanisms.

LSTM (Long Short-Term Memory):

Maintains two states: hidden state $\mathbf{h}_t$ and cell state $\mathbf{c}_t$ (the "memory highway").

Three gates (all sigmoid → values in $[0,1]$):

• Forget gate: $\mathbf{f}_t = \sigma(W_f[\mathbf{h}_{t-1}, \mathbf{x}_t])$ — what to erase from memory
• Input gate: $\mathbf{i}_t = \sigma(W_i[\mathbf{h}_{t-1}, \mathbf{x}_t])$ — what new info to store
• Output gate: $\mathbf{o}_t = \sigma(W_o[\mathbf{h}_{t-1}, \mathbf{x}_t])$ — what to output

Cell update: $\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tanh(W_c[\mathbf{h}_{t-1}, \mathbf{x}_t])$ Hidden state: $\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$

Why it works: the cell state flows through with mostly multiplicative interactions by the forget gate. When $\mathbf{f}_t \approx 1$, information passes through unchanged — gradient flows unimpeded.

GRU (Gated Recurrent Unit):

• Simpler: merges cell and hidden state, uses 2 gates (reset + update) instead of 3
• Similar performance to LSTM in practice, fewer parameters

Why transformers replaced RNNs:

• RNNs process sequentially → can't parallelize
• Even LSTMs struggle with very long sequences (>500 steps)
• Self-Attention gives direct connections between any two positions

See also: Recurrent Neural Networks, Self-Attention