Residual Connections

A residual (skip) connection adds the input of a layer to its output:

where $F$ is the layer's transformation (attention, feed-forward, etc.).

The residual stream view: the input flows through a "stream" and each layer reads from and writes to it. The stream carries the original signal forward; layers contribute incremental updates.

Why it works:

1. Gradient flow: during Backpropagation, $\frac{\partial \mathbf{x}_{l+1}}{\partial \mathbf{x}_l} = I + \frac{\partial F}{\partial \mathbf{x}_l}$. The identity term $I$ provides a gradient highway — even if $\frac{\partial F}{\partial \mathbf{x}_l}$ vanishes, the gradient passes through unimpeded
2. Easier optimization: the network only needs to learn the residual $F(\mathbf{x})$ (deviation from identity), which is often simpler
3. Enables depth: ResNets (152 layers), transformers (96+ layers) would be untrainable without skip connections

In transformers: every sub-layer (attention, feed-forward) has a residual connection:

Pre-norm vs post-norm: modern transformers apply LayerNorm before the sub-layer (pre-norm) for more stable training.

See also: Backpropagation, Batch Normalization, Self-Attention