Layer Normalization

Layer normalization normalizes across the feature dimension for each individual sample:

\text{LayerNorm}(\mathbf{x}) = \gamma \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta

where $\mu$ and $\sigma^2$ are the mean and variance computed over the features of a single input, and $\gamma, \beta$ are learned scale and shift parameters.

LayerNorm vs BatchNorm:

	BatchNorm	LayerNorm
Normalizes over	Batch dimension	Feature dimension
Depends on batch	Yes (problematic for small batches)	No
At inference	Uses running statistics	Same as training
Dominant in	CNNs	Transformers

Why transformers use LayerNorm:

Sequences have variable length → batch statistics across positions are meaningless
No dependence on batch size → works with any batch, including batch size 1
Stabilizes the residual stream by keeping activations on a consistent scale

Pre-Norm vs Post-Norm:

Post-Norm (original transformer): $x + \text{LayerNorm}(\text{Sublayer}(x))$ — harder to train deep models
Pre-Norm (GPT-2+, standard now): $x + \text{Sublayer}(\text{LayerNorm}(x))$ — more stable gradients, enables deeper models

RMSNorm — a simplified variant that skips the mean subtraction and the bias $\beta$ , dividing only by the root-mean-square:

\text{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\text{RMS}(\mathbf{x}) + \epsilon} \odot \gamma, \qquad \text{RMS}(\mathbf{x}) = \sqrt{\tfrac{1}{D}\textstyle\sum_{i=1}^{D} x_i^2}

Forcing unit RMS destroys learned scale, so the per-dimension $\gamma \in \mathbb{R}^D$ gives it back: $\gamma_i > 1$ amplifies dimension $i$ , $\gamma_i \approx 0$ kills it. Cheaper than LayerNorm (no mean, no $\beta$ ) and equally stable in practice — used in LLaMA and other modern LLMs.

Layer Normalization

Linked from