Linear Regression

Linear regression models the relationship $y = \mathbf{w}^\top\mathbf{x} + b$ (or $y = X\mathbf{w}$ with bias absorbed).

Two ways to solve:

Closed-form (Normal Equations):

• Direct solution, no iteration
• Equivalent to MLE assuming Gaussian noise
• $O(d^3)$ — impractical for very high dimensions

Gradient descent:

• Scales to large datasets via Stochastic Gradient Descent
• Same solution as closed-form (MSE is convex → unique minimum)

Loss function: MSE = $\frac{1}{n}\sum(y_i - \hat{y}_i)^2$ — assumes Gaussian noise, equivalent to MLE.

Adding Regularization:

• Ridge (L2): $(X^\top X + \lambda I)^{-1}X^\top y$ — MAP with Gaussian prior
• Lasso (L1): no closed form, promotes sparsity

See also: Logistic Regression, Bias-Variance Tradeoff