# topic/math/linear-algebra

Cross Product

The cross product is defined only in and and produces a vector (unlike the Dot Product which produces a scalar):

Determinant and Inverse

The determinant of a square matrix is a scalar that captures how the transformation scales volume:

Dot Product

The dot product (inner product) of two vectors can be understood through two equivalent views:

Eigendecomposition

An eigenvector of matrix is a nonzero vector whose direction is unchanged by the transformation: , where is the eigenvalue.

Gaussian Elimination

Gaussian elimination transforms a matrix into row echelon form (REF) using elementary row operations to solve linear systems .

Linear Algebra

Linear algebra studies vector spaces and linear maps between them. The useful mental model is: vectors live in spaces, bases give coordinates, matrices represent transformations…

Linear Transformations

A linear transformation satisfies . Every linear transformation can be represented as multiplication by a matrix , and every matrix defines one.

LoRA

LoRA (Low-Rank Adaptation) makes fine-tuning large models practical by training only small low-rank matrices.

Matrix Multiplication

Matrix multiplication can be understood through three equivalent views:

Norms and Distance Metrics

A norm measures the "size" of a vector. Norms underlie nearly every loss function, regularizer, and similarity measure in ML.

Orthogonality and Projections

Two vectors are orthogonal if . An orthonormal set has all vectors mutually orthogonal with unit length.

Positive Definite Matrices

A symmetric matrix is positive definite (PD) if for all nonzero . Positive semi-definite (PSD) allows .

Power Iteration

Power iteration is a simple algorithm to find the dominant eigenvalue (largest in absolute value) and its eigenvector.

Principal Component Analysis (PCA)

PCA finds the directions of maximum variance in data and projects onto them for dimensionality reduction.

Rank and Null Space

For a matrix ( ):

Singular Value Decomposition (SVD)

Every matrix (any shape) can be decomposed as :

Vector Spaces and Basis

A vector space over is a set of vectors closed under addition and scalar multiplication. is the canonical example.