Classical Machine Learning

Classical machine learning studies how models generalize from data without relying on large neural networks. The useful mental model is: choose a hypothesis class, define a loss, estimate parameters from training data, and evaluate whether the learned pattern generalizes.

Core sequence:

1. Inductive Bias - the assumptions that make learning possible.
2. Curse of Dimensionality - why high-dimensional spaces make naive learning and search difficult.
3. Loss Functions - objectives that convert prediction errors into quantities to minimize.
4. Evaluation Metrics - task-level measurements of model behavior.
5. Cross-Validation - estimates out-of-sample performance by rotating train/validation splits.
6. Bias-Variance Tradeoff - decomposes error into underfitting, sensitivity to data, and irreducible noise.
7. Regularization - constrains models to improve generalization.
8. Linear Regression - predicts continuous targets with a linear function.
9. Logistic Regression - models class probabilities with a linear score and sigmoid or softmax link.
10. K-Nearest Neighbors - predicts from nearby examples under a distance metric.
11. Random Forest - ensembles decision trees to reduce variance.
12. Approximate Nearest Neighbor Search - finds near matches efficiently in large vector collections.

How the pieces fit:

• Inductive bias restricts what the model can learn.
• Losses define the training objective; metrics define the external judgment.
• Cross-validation estimates whether the model generalizes.
• Bias-variance and regularization explain underfitting, overfitting, and capacity control.
• Linear and logistic regression are parametric baselines.
• KNN, random forests, and approximate nearest neighbor search use local or ensemble structure rather than a single global linear function.

Core equations to keep active:

• Empirical risk: $\hat{R}(f) = \frac{1}{n}\sum_i \ell(f(x_i), y_i)$
• Regularized risk: $\hat{R}_\lambda(f) = \frac{1}{n}\sum_i \ell(f(x_i), y_i) + \lambda \Omega(f)$
• Bias-variance pattern: $\mathbb{E}[(y-\hat{f}(x))^2] = \mathrm{bias}^2 + \mathrm{variance} + \sigma^2$
• Linear regression: $\hat{y} = \mathbf{w}^\top\mathbf{x} + b$
• Normal equations: $\hat{\mathbf{w}} = (X^\top X)^{-1}X^\top \mathbf{y}$
• Logistic regression: $p(y=1|x) = \sigma(\mathbf{w}^\top x + b)$
• KNN prediction: $\hat{y}(x) = \mathrm{majority}\{y_i: x_i \in N_k(x)\}$
• Random forest prediction: $\hat{f}(x) = \frac{1}{T}\sum_{t=1}^T f_t(x)$ for regression.

See also: Probability, Statistics Fundamentals, Linear Algebra