Systems and Scaling

Systems and scaling study how model training and inference behave on real hardware. The useful mental model is: performance is limited by compute, memory, communication, and numerical precision, and modern ML systems are mostly about balancing those constraints.

Core sequence:

1. Floating Point and Quantization - represents real numbers with finite precision and reduces memory or compute cost with lower precision.
2. Memory Hierarchy and IO-Awareness - explains why moving data can dominate arithmetic.
3. GPU Architecture and CUDA - describes the parallel hardware model used for neural network workloads.
4. Distributed Training Strategies - splits training across devices through data, tensor, pipeline, or expert parallelism.
5. Computational Complexity of Attention - shows why transformer context length stresses memory and compute.

How the pieces fit:

• Floating point and quantization define what numbers the system can represent cheaply.
• Memory hierarchy determines whether kernels are compute-bound or bandwidth-bound.
• GPUs provide massive parallelism, but only when work is organized to fit the hardware.
• Distributed training adds communication as a first-class bottleneck.
• Attention complexity is the LLM-specific scaling pressure that links algorithms to systems.

Core equations to keep active:

• Approximate roofline bound: $\mathrm{time} \geq \max\left(\frac{\mathrm{FLOPs}}{\mathrm{peak\ FLOPs}}, \frac{\mathrm{bytes}}{\mathrm{memory\ bandwidth}}\right)$
• Arithmetic intensity: $\frac{\mathrm{FLOPs}}{\mathrm{bytes\ moved}}$
• Attention score size: $n \times n$ for sequence length $n$.
• KV-cache memory pattern: $O(L n d)$ for layers $L$, context length $n$, and hidden width or per-layer cache width $d$.
• Data-parallel effective batch: $B_{\mathrm{global}} = B_{\mathrm{per\ device}} \times N_{\mathrm{devices}} \times N_{\mathrm{accumulation}}$
• Quantization scale pattern: $x \approx s(q - z)$ for scale $s$, integer value $q$, and zero point $z$.

See also: Computer Science Foundations, Transformers and LLMs