Causal Masking

Causal masking restricts Self-Attention so that each position can only attend to itself and earlier positions — never future tokens.

Implementation: before the softmax in attention, set all entries above the diagonal of the $QK^\top$ matrix to $-\infty$:

After softmax, $-\infty$ entries become 0 → zero attention to future positions.

Why it's needed:

• Decoder-only models (GPT) are trained with next-token prediction → at position $t$, the model must predict token $t+1$ using only tokens $1, \dots, t$
• Without masking, the model could "cheat" by looking at the answer
• During generation, future tokens don't exist yet — masking during training simulates this

Types of transformers:

• Encoder-only (BERT): no causal mask, full bidirectional attention → good for understanding
• Decoder-only (GPT): causal mask → Autoregressive Generation
• Encoder-decoder (T5, original transformer): encoder is bidirectional, decoder is causal with cross-attention to encoder

Efficiency: causal masking means attention is a triangular operation. FlashAttention and other optimized kernels exploit this structure.

See also: Self-Attention, Autoregressive Generation, Multi-Head Attention