Linear Algebra

Linear algebra studies vector spaces and linear maps between them. The useful mental model is: vectors live in spaces, bases give coordinates, matrices represent transformations, and decompositions reveal which directions are preserved, collapsed, or amplified.

Core sequence:

1. Vector Spaces and Basis - the setting. A vector space is closed under addition and scalar multiplication; a basis is a minimal coordinate system for the space.
2. Dot Product - turns vectors into geometry. It measures alignment, defines orthogonality, and gives the algebra behind matrix multiplication.
3. Norms and Distance Metrics - measure vector size and pairwise distance. Norms turn algebraic objects into quantities that can be optimized.
4. Matrix Multiplication - combines dot products, linear combinations, and transformation composition into one operation.
5. Linear Transformations - the function view of matrices: $T(\mathbf{x}) = A\mathbf{x}$ maps one vector space into another while preserving linear structure.
6. Orthogonality and Projections - decomposes vectors into components along and perpendicular to subspaces. This is the geometry behind least squares and PCA.
7. Gaussian Elimination - the computational tool for solving $A\mathbf{x} = \mathbf{b}$, finding pivots, and exposing rank.
8. Determinant and Inverse - square-matrix diagnostics. The determinant measures volume scaling; an inverse exists exactly when the transformation does not collapse a dimension.
9. Rank and Null Space - describes what information a matrix preserves and destroys. Rank is the output dimension; null space is the set of inputs mapped to zero.
10. Eigendecomposition - finds directions unchanged by a square transformation except for scaling: $A\mathbf{v} = \lambda \mathbf{v}$.
11. Singular Value Decomposition (SVD) - generalizes the geometric decomposition to any matrix: $A = U\Sigma V^\top$.
12. Positive Definite Matrices - symmetric matrices whose quadratic form is always positive. These encode curvature, covariance, and stable optimization geometry.
13. Principal Component Analysis (PCA) - applies eigendecomposition or SVD to find low-dimensional directions that preserve maximum variance.

How the pieces fit:

• Vector spaces and bases define the objects being manipulated.
• Dot products, norms, orthogonality, and projections add geometry.
• Matrices and matrix multiplication represent transformations and their composition.
• Gaussian elimination, determinants, inverses, rank, and null spaces diagnose solvability and information loss.
• Eigendecomposition and SVD expose the important directions of a transformation.
• Positive definite matrices connect linear algebra to curvature, covariance, and optimization.
• PCA is the downstream ML pattern: choose a lower-dimensional basis by using the geometry of variance.

Core equations to keep active:

• Dot product: $\mathbf{a} \cdot \mathbf{b} = \sum_i a_i b_i = \|\mathbf{a}\|\|\mathbf{b}\|\cos\theta$
• Euclidean norm: $\|\mathbf{x}\|_2 = \sqrt{\sum_i x_i^2}$
• Matrix product entry: $(AB)_{ij} = \sum_k A_{ik}B_{kj}$
• Projection: $\mathrm{proj}_{\mathbf{a}}\mathbf{b} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}}\mathbf{a}$
• Rank-nullity: $\mathrm{rank}(A) + \mathrm{nullity}(A) = n$
• Eigenvector equation: $A\mathbf{v} = \lambda\mathbf{v}$
• SVD: $A = U\Sigma V^\top$
• Positive definite test: $\mathbf{x}^\top A\mathbf{x} > 0$ for all nonzero $\mathbf{x}$
• PCA covariance: $C = \frac{1}{n}\tilde{X}^\top\tilde{X}$