Floating Point and Quantization

3 min read

# topic/cs/system

Numbers in hardware have finite precision. Choosing the right format trades off range, precision, memory, and speed.

Common formats:

FormatBitsExponentMantissaRangeUse
FP3232823±3.4×1038\pm 3.4 \times 10^{38}Master weights, loss scaling
FP1616510±65,504\pm 65,504Training with loss scaling
BF161687±3.4×1038\pm 3.4 \times 10^{38}Training default on modern GPUs
FP8 (E4M3)843±448\pm 448Forward pass in latest hardware
INT88——[−128,127][-128, 127]Post-training quantization
INT44——[−8,7][-8, 7]Aggressive inference quantization

BF16 vs FP16: BF16 has the same exponent range as FP32 (no overflow issues) but less precision. FP16 has more precision but can overflow → requires loss scaling. BF16 is the standard for training.

Mixed-precision training:

  1. Keep FP32 master copy of weights
  2. Cast to BF16/FP16 for forward and backward pass (2x speed on Tensor Cores, half memory)
  3. Accumulate gradients in FP32
  4. Update master weights in FP32

Gradients must land in FP32 because BF16 has fine resolution near 0 (it can represent a tiny gradient like 0.0001) but cannot represent the updated weight (1.0001) — adding a small gradient to a large weight would be lost to rounding. Matmuls tolerate rounding noise, so BF16 forward/backward is fine.

# autocast: weights stay FP32, ops selectively run in BF16 (matmul) or FP32 (softmax, layernorm)
with torch.autocast(device_type='cuda', dtype=torch.bfloat16):
    output = model(x)
 
# or load weights directly in BF16 (simplest, great for inference):
model = AutoModel.from_pretrained(name, torch_dtype=torch.bfloat16)
 
# 8-bit weights via bitsandbytes (weights INT8, activations FP16):
model = AutoModel.from_pretrained(name, load_in_8bit=True)

Quantization for inference:

  • Post-training quantization (PTQ) — quantize a trained model. Calibrate scale/zero-point on a small dataset
  • Quantization-aware training (QAT) — simulate quantization during training via straight-through estimator
  • GPTQ, AWQ, GGML — specialized LLM quantization methods
  • Going from FP16 to INT4 → 4x memory reduction, enabling large models on consumer GPUs

Numerical pitfalls in ML:

  • Subtracting nearly equal numbers → catastrophic cancellation
  • Softmax overflow → subtract max⁡(z)\max(\mathbf{z}) first (log-sum-exp trick)
  • Gradient underflow in FP16 → dynamic loss scaling multiplies loss by a large factor, scales gradients back down after

See also: GPU Architecture and CUDA, Memory Hierarchy and IO-Awareness, LoRA

Linked from