Principal Component Analysis (PCA)

PCA finds the directions of maximum variance in data and projects onto them for dimensionality reduction.

Via eigendecomposition:

1. Center the data: $\tilde{X} = X - \bar{X}$
2. Compute covariance matrix: $C = \frac{1}{n}\tilde{X}^\top\tilde{X}$
3. Eigendecompose $C$: eigenvectors = principal directions, eigenvalues = variance along each direction
4. Project onto top $k$ eigenvectors

Via SVD (preferred numerically):

1. Center data matrix $\tilde{X}$ ($n \times d$)
2. Compute $\tilde{X} = U\Sigma V^\top$
3. Columns of $V$ are principal components; $\Sigma^2/n$ gives variances
4. Projected data = $U_k\Sigma_k$ (top $k$ components)

Key insight: PCA finds the subspace that preserves the most variance (equivalently, minimizes reconstruction error).

Limitations:

• Only captures linear relationships
• Sensitive to feature scaling — always standardize first
• Assumes variance = importance

See also: Eigendecomposition, Singular Value Decomposition (SVD), Bias-Variance Tradeoff