Vector Spaces and Basis

A vector space $V$ over $\mathbb{R}$ is a set of vectors closed under addition and scalar multiplication. $\mathbb{R}^n$ is the canonical example.

Core definitions:

• Linear combination: $c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \dots + c_k\mathbf{v}_k$
• Span: the set of all linear combinations of a set of vectors
• Linear independence: no vector in the set can be written as a linear combination of the others
• Basis: a linearly independent set that spans the entire space
• Dimension: the number of vectors in any basis (unique for a given space)

Subspaces are vector spaces living inside a larger space. The most important ones:

• Column space $C(A)$ — what outputs $A\mathbf{x}$ can produce → range of the transformation
• Null space $N(A)$ — what inputs $A\mathbf{x} = \mathbf{0}$ annihilates → see Rank and Null Space
• Row space — span of the rows of $A$

Why this matters for ML:

• A neural network layer maps between vector spaces: $\mathbb{R}^n \to \mathbb{R}^m$
• Embeddings live in learned vector spaces where directions encode meaning
• Principal Component Analysis (PCA) finds the best low-dimensional basis for the data
• Rank of a weight matrix = dimension of its column space → if rank-deficient, the layer has redundant capacity (LoRA exploits this)

See also: Rank and Null Space, Linear Transformations, Orthogonality and Projections