Weight Initialization

Wrong initialization causes vanishing or exploding gradients before training even begins.

The problem: if weights are too large, activations saturate or explode. Too small, signals vanish through layers.

Xavier/Glorot initialization (for sigmoid/tanh):

• Preserves variance of activations and gradients through layers
• Derived by requiring $\text{Var}(\text{output}) = \text{Var}(\text{input})$

Kaiming/He initialization (for ReLU):

• Accounts for ReLU zeroing out half the activations (factor of 2)
• Default for ReLU networks

Biases: typically initialized to zero.

Why this matters:

• A 50-layer network with $\text{Var}(W) = 1$ would have activations $\sim 2^{50}$ or $\sim 2^{-50}$ — unusable
• Correct initialization keeps the "signal" flowing through the network
• Batch Normalization and Residual Connections provide additional stabilization

See also: Backpropagation, Activation Functions