The expected test error of a model decomposes into three terms:
Bias: error from wrong assumptions in the model. High bias = underfitting.
- Linear model on nonlinear data → high bias
- Doesn't capture the true relationship
Variance: error from sensitivity to fluctuations in training data. High variance = overfitting.
- Complex model memorizes training noise
- Performs well on training data, poorly on test data
The tradeoff:
- Simple models → high bias, low variance (underfit)
- Complex models → low bias, high variance (overfit)
- Sweet spot depends on dataset size and noise level
Controlling the tradeoff:
- Regularization — adds bias, reduces variance
- More training data — reduces variance without adding bias
- Cross-Validation — estimates where you are on the tradeoff curve
- Model selection — choose complexity appropriate for data size
- Ensemble methods (Random Forest) — reduce variance by averaging
In deep learning: overparameterized networks (more params than data) should overfit badly but often don't — "double descent" phenomenon challenges the classical picture.
See also: Regularization, Cross-Validation, Evaluation Metrics