Norms and Distance Metrics

A norm $\|\mathbf{x}\|$ measures the "size" of a vector. Norms underlie nearly every loss function, regularizer, and similarity measure in ML.

Common norms ( $L^p$ family):

Norm	Formula	Use in ML
$L^1$ (Manhattan)	$\sum_i \\|x_i\\|$	Lasso regression, sparse solutions
$L^2$ (Euclidean)	$\sqrt{\sum_i x_i^2}$	Default distance, Ridge regression, weight decay
$L^\infty$ (Max)	$\max_i \\|x_i\\|$	Adversarial robustness (perturbation budgets)
$L^0$ (pseudo-norm)	count of nonzero entries	Sparsity (NP-hard to optimize, $L^1$ is the convex relaxation)

Frobenius norm for matrices: $\|A\|_F = \sqrt{\sum_{ij} a_{ij}^2}$ — the $L^2$ norm of a matrix flattened into a vector.

Key distance metrics:

Euclidean distance: $d(\mathbf{a}, \mathbf{b}) = \|\mathbf{a} - \mathbf{b}\|_2$
Cosine similarity: $\frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}$ — measures angle, ignores magnitude. Dominant in embeddings and retrieval
Mahalanobis distance: accounts for covariance structure — Euclidean distance after whitening

Why this matters everywhere:

Regularization is just adding a norm penalty: $L^1$ → sparsity, $L^2$ → small weights
Loss Functions are often norms of the residual: MSE = squared $L^2$ , MAE = $L^1$
Cosine similarity is the scoring function inside Self-Attention (after normalization by $\sqrt{d_k}$ )

Linked from