Positional Encoding

Transformers have no built-in notion of order — Self-Attention is permutation-equivariant. Positional encodings inject position information.

Sinusoidal (original "Attention Is All You Need"):

PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d})

PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d})

Each dimension oscillates at a different frequency. Properties:

Deterministic, no learnable parameters
Can extrapolate to longer sequences (in theory)
Relative positions are representable as linear functions of the encoding

Learned positional embeddings: add a learnable embedding vector for each position (up to max length). Used in GPT-2, BERT. Simple but can't extrapolate beyond training length.

Rotary Position Embeddings (RoPE): encode relative position by rotating Q and K vectors (values are left untouched — position only affects which tokens attend, not what is passed). For a query at position $m$ and key at position $n$ , rotating both makes the dot product depend only on the relative offset $m - n$ :

\langle R_{m\theta}\mathbf{q}, R_{n\theta}\mathbf{k}\rangle = \mathbf{q}^\top R_{(n-m)\theta}\,\mathbf{k}

because $R_\alpha^\top R_\beta = R_{\beta - \alpha}$ . The head dimension is split into $H/2$ pairs, each rotated at frequency $\theta_i = \Theta^{-2i/H}$ (base $\Theta$ usually 10,000). Low-frequency pairs rotate slowly → encode long-range position; high-frequency pairs enable local discrimination.

In practice, precompute $\cos(m\theta_i), \sin(m\theta_i)$ and apply element-wise rather than building rotation matrices:

# precompute once
positions = torch.arange(max_seq_len)
thetas = theta ** (-torch.arange(0, d_k, 2) / d_k)   # (d_k // 2)
angles = positions.unsqueeze(-1) * thetas.unsqueeze(0)   # (max_seq_len, d_k // 2)
 
# apply: split into even/odd pairs, rotate, re-interleave
x_pairs = x.reshape(*x.shape[:-1], -1, 2)
x_even, x_odd = x_pairs[..., 0], x_pairs[..., 1]
x_out_even = x_even * cos - x_odd * sin
x_out_odd  = x_even * sin + x_odd * cos
x_out = torch.stack([x_out_even, x_out_odd], dim=-1).flatten(start_dim=-2)

Used in modern LLMs (LLaMA, GPT-NeoX). Better length generalization.

ALiBi (Attention with Linear Biases): add a position-dependent bias to attention scores. No parameters, good extrapolation.

Key point: without positional encoding, "The cat sat on the mat" and "mat the on sat cat The" would produce identical attention patterns — the model couldn't distinguish word order.

Positional Encoding

Linked from