Regularization

1 min read

# topic/ai-ml/classical

Regularization adds a penalty to the loss function to prevent overfitting by constraining model complexity.

L2 regularization (Ridge / Weight Decay):

Ltotal=Ldata+λ2∥w∥2L_{\text{total}} = L_{\text{data}} + \frac{\lambda}{2}\|\mathbf{w}\|^2
  • Shrinks all weights toward zero proportionally
  • Equivalent to MAP with Gaussian prior
  • In Adam: use decoupled weight decay (AdamW) for correct behavior

L1 regularization (Lasso):

Ltotal=Ldata+λ∥w∥1L_{\text{total}} = L_{\text{data}} + \lambda\|\mathbf{w}\|_1
  • Drives some weights exactly to zero → automatic feature selection / sparsity
  • Equivalent to MAP with Laplace prior

Why regularization prevents overfitting:

  • Constrains the hypothesis space (smaller weights → simpler functions)
  • Reduces model capacity, shifting the Bias-Variance Tradeoff toward lower variance at the cost of slightly more bias
  • The regularization strength λ\lambda controls this tradeoff

Other forms of regularization in deep learning:

  • Dropout (random neuron removal during training)
  • Batch Normalization (implicit regularization)
  • Data augmentation
  • Early stopping

See also: Maximum A Posteriori Estimation, Bias-Variance Tradeoff, Linear Regression

Linked from