Batch Normalization

Batch normalization normalizes activations within each mini-batch to stabilize and accelerate training.

Algorithm (for each feature dimension in a layer):

1. Compute mini-batch mean: $\mu_B = \frac{1}{B}\sum_{i=1}^B x_i$
2. Compute mini-batch variance: $\sigma_B^2 = \frac{1}{B}\sum_{i=1}^B (x_i - \mu_B)^2$
3. Normalize: $\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$
4. Scale and shift (learnable): $y_i = \gamma\hat{x}_i + \beta$

Placement: typically after the linear transformation, before the activation function.

Why it helps:

• Reduces internal covariate shift — each layer sees stable input distributions
• Allows higher learning rates without divergence
• Acts as implicit Regularization (mini-batch noise)
• Smooths the loss landscape → easier optimization

At inference: use running averages of $\mu$ and $\sigma^2$ computed during training (not batch statistics).

Variants:

• Layer Normalization — normalizes across features (not batch). Used in transformers because it works with variable sequence lengths and doesn't depend on batch size
• Group Normalization — between BatchNorm and LayerNorm
• RMSNorm — simplified LayerNorm without centering. Used in modern LLMs

See also: Weight Initialization, Backpropagation, Residual Connections