Orthogonality and Projections

Two vectors are orthogonal if $\mathbf{a} \cdot \mathbf{b} = 0$. An orthonormal set has all vectors mutually orthogonal with unit length.

Projection of $\mathbf{b}$ onto $\mathbf{a}$:

Projection onto a subspace spanned by orthonormal columns $Q$: $\text{proj} = QQ^\top\mathbf{b}$.

Gram-Schmidt process: takes any basis and produces an orthonormal one — iteratively subtract projections onto previous vectors and normalize.

Key properties of orthogonal matrices ($Q^\top Q = I$):

• They preserve lengths and angles (rotations and reflections)
• $Q^{-1} = Q^\top$ — trivially cheap to invert
• Condition number is 1 → numerically stable

Where this appears in ML:

• Principal Component Analysis (PCA) — finds orthogonal directions of maximum variance
• Singular Value Decomposition (SVD) — decomposes any matrix into orthogonal rotations + scaling
• Attention — queries and keys live in spaces where dot product measures alignment; orthogonal means "unrelated"
• Weight orthogonalization — initialization and regularization technique to preserve gradient norms
• Least squares — the residual $\mathbf{b} - A\hat{\mathbf{x}}$ is orthogonal to the column space of $A$

See also: Dot Product, Vector Spaces and Basis, Singular Value Decomposition (SVD)