GPUs achieve massive parallelism through thousands of simple cores executing the same instruction on different data (SIMT — Single Instruction, Multiple Threads).
Hardware hierarchy (NVIDIA):
- Streaming Multiprocessors (SMs) — the compute units (e.g., A100 has 108 SMs)
- CUDA cores — ALUs within each SM (thousands total)
- Tensor Cores — specialized units for mixed-precision matrix multiply-accumulate (the reason training is fast)
- Warps — groups of 32 threads that execute in lockstep. Warp divergence (branching) kills performance
Memory hierarchy:
- Registers — per-thread, fastest (~TB/s effective bandwidth)
- Shared memory / L1 cache — per-SM, ~19 TB/s on A100 (192 KB per SM)
- L2 cache — shared across SMs, ~40 MB on A100
- HBM (global memory) — 80 GB on A100, ~2 TB/s bandwidth. This is the bottleneck → see Memory Hierarchy and IO-Awareness
Why matrix multiplication is perfectly suited for GPUs:
- Highly regular, parallelizable computation
- High arithmetic intensity (FLOPs per byte loaded) — compute on data
- Tensor Cores do 4x4 matrix multiply in one cycle
Key concepts for ML:
- Occupancy — fraction of SM resources utilized. Low occupancy = wasted parallelism
- Memory coalescing — adjacent threads should access adjacent memory addresses
- Kernel fusion — combining multiple operations into one GPU kernel to avoid memory round-trips (this is what FlashAttention does)
- Mixed precision — use fp16/bf16 for compute, fp32 for accumulation → 2x speed, half memory → Floating Point and Quantization
Software stack: CUDA → cuDNN / cuBLAS → PyTorch/JAX. Most ML researchers never write CUDA directly but understanding the hardware explains why certain operations are fast or slow.
See also: Memory Hierarchy and IO-Awareness, Floating Point and Quantization, Distributed Training Strategies