Scaling Laws

Scaling laws describe how LLM performance (loss) improves predictably as you increase model size, data, and compute.

Kaplan et al. (2020) power laws:

where $N$ = parameters, $D$ = dataset size, $C$ = compute (FLOPs), and $L$ = cross-entropy loss.

Key finding: loss decreases as a smooth power law in each factor, spanning many orders of magnitude. No ceiling in sight (so far).

Chinchilla scaling (Hoffmann et al., 2022):

• For a given compute budget, there is an optimal ratio of model size to data
• Chinchilla law: tokens should scale linearly with parameters (~20 tokens per parameter)
• Previous models (GPT-3) were undertrained — too large for the data they saw
• Led to smaller, better-trained models (Chinchilla, LLaMA)

Fitting a scaling law: the loss-vs-compute curve needs an irreducible loss term $\mathcal{L}_\infty$ (the entropy of the data) — otherwise the fit implies $\mathcal{L} \to 0$ as $C \to \infty$. Taking logs linearizes it:

Optimal hyperparameters scale too, e.g. $\text{LR}(C) = \beta C^{-\alpha}$. Fitting is done by least squares (minimizing squared residuals $S = \sum_i (y_i - f(x_i))^2$): linear least squares has the closed form $\beta = (X^\top X)^{-1}X^\top y$; the power-law fit above, once linearized, is solved the same way (otherwise non-linear least squares via iterative refinement).

Maximal update parameterization (μP): hyperparameters tuned at small width don't transfer to large width because standard parameterization gives updates of inconsistent magnitude as width grows. μP adjusts per-layer initialization and learning rates so the magnitude of updates relative to weights stays constant across widths — letting you tune cheaply at small scale and transfer to large.

Practical implications:

• Performance is predictable from small-scale experiments → can plan large training runs
• More compute always helps (if allocated correctly between N and D)
• Emergent abilities may appear at scale (e.g., in-context learning, chain-of-thought reasoning)

Open questions: do scaling laws hold for reasoning? For alignment? For specialized domains?

See also: Pretraining, Supervised Fine-Tuning