Singular Value Decomposition (SVD)

Every matrix $M$ (any shape) can be decomposed as $M = U\Sigma V^\top$:

• $U$ — orthogonal matrix of left singular vectors (column space directions)
• $\Sigma$ — diagonal matrix of singular values $\sigma_1 \geq \sigma_2 \geq \dots \geq 0$
• $V^\top$ — orthogonal matrix of right singular vectors (row space directions)

Intuition: any linear transformation = rotate ($V^\top$) → scale ($\Sigma$) → rotate ($U$).

Low-rank approximation: keep only the top $k$ singular values → best rank-$k$ approximation in Frobenius norm (Eckart-Young theorem). This is the mathematical foundation of:

• Dimensionality reduction
• Image compression
• LoRA — adds low-rank updates $\Delta W = BA$ where $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times d}$, $r \ll d$

Relation to eigendecomposition:

• Singular values of $M$ = square roots of eigenvalues of $M^\top M$
• SVD works for any matrix; Eigendecomposition requires square (and ideally diagonalizable) matrices

PCA connection: Principal Component Analysis (PCA) via SVD of the centered data matrix gives principal components without forming the covariance matrix explicitly.

See also: Rank and Null Space