Dropout

Dropout randomly sets each neuron's output to zero with probability $p$ during training:

The $1/(1-p)$ scaling (inverted dropout) ensures expected output magnitude stays the same, so no adjustment is needed at inference time.

Why it works:

• Prevents co-adaptation — neurons can't rely on specific other neurons being present, forcing redundant representations
• Implicit ensemble — each forward pass uses a different random subnetwork; inference averages over all $2^n$ subnetworks
• Acts as a stochastic regularizer — similar effect to $L^2$ penalty in linear models

Practical details:

• Common values: $p = 0.1$–$0.5$ (higher = more regularization)
• Applied after activation functions, typically before the next linear layer
• Disabled at inference — always use full network at test time
• Not used with Batch Normalization in the same block (they interact poorly)

In modern architectures:

• Transformers use dropout on attention weights and after feed-forward layers
• Large pretrained models (GPT-3+) often train with dropout = 0 and rely on data scale instead
• DropPath (stochastic depth) — drops entire residual blocks; used in vision transformers

See also: Regularization, Batch Normalization, Residual Connections