# topic/cs/system

Distributed Training Strategies

When a model or dataset is too large for a single GPU, training must be distributed across multiple devices.

Floating Point and Quantization

Numbers in hardware have finite precision. Choosing the right format trades off range, precision, memory, and speed.

GPU Architecture and CUDA

GPUs achieve massive parallelism through thousands of simple cores executing the same instruction on different data (SIMT — Single Instruction, Multiple Threads).

Memory Hierarchy and IO-Awareness

Modern hardware is memory-bound, not compute-bound for most ML operations. Understanding the memory hierarchy is the key to writing fast code.

Systems and Scaling

Systems and scaling study how model training and inference behave on real hardware. The useful mental model is: performance is limited by compute, memory, communication, and…