LoRA

LoRA (Low-Rank Adaptation) makes fine-tuning large models practical by training only small low-rank matrices.

Core idea: instead of updating the full weight matrix $W \in \mathbb{R}^{d \times d}$, freeze $W$ and add a low-rank update:

where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times d}$, and $r \ll d$.

Connection to Singular Value Decomposition (SVD) : the update $\Delta W = BA$ has rank at most $r$. SVD tells us that weight updates during fine-tuning tend to be low-rank — most of the change lives in a low-dimensional subspace. LoRA exploits this by parameterizing the update in that subspace directly.

Parameter savings:

• Full fine-tuning: $d \times d$ parameters per layer
• LoRA: $d \times r + r \times d = 2dr$ parameters per layer
• With $d = 4096$ and $r = 16$: $16M$ vs $131K$ — 100x fewer trainable parameters

Training:

• $A$ initialized with random Gaussian, $B$ initialized to zero (so $\Delta W = 0$ at start)
• Only $A$ and $B$ are updated; base model stays frozen
• At inference: merge $W + BA$ — no additional latency

Practical use:

• Supervised Fine-Tuning and Direct Preference Optimization with LoRA via PEFT + TRL libraries
• Multiple LoRA adapters can be swapped on the same base model
• QLoRA: combine with quantization (4-bit base model) for even cheaper fine-tuning

See also: Singular Value Decomposition (SVD) , Supervised Fine-Tuning