Entropy and Cross-Entropy

Entropy $H(P)$ measures the average surprise (information content) of a distribution:

• Uniform distribution has maximum entropy (most uncertain)
• A delta distribution (all mass on one outcome) has zero entropy
• Units: bits (log base 2) or nats (natural log)

Cross-entropy $H(P, Q)$ measures the average bits needed to encode samples from $P$ using a code optimized for $Q$:

Since $H(P)$ is constant w.r.t. model parameters, minimizing cross-entropy = minimizing KL Divergence from the true distribution.

Why cross-entropy is THE classification loss:

• For a true label $y$ and predicted distribution $\hat{p}$:
  • Binary: $L = -[y\log\hat{p} + (1-y)\log(1-\hat{p})]$
  • Multiclass: $L = -\log\hat{p}_y$ (negative log-likelihood of the correct class)
• This is equivalent to Maximum Likelihood Estimation of the model parameters
• It heavily penalizes confident wrong predictions (because $-\log(0) \to \infty$)

Language-model loss in practice — shift logits/labels by one (predict token $t{+}1$ from $\leq t$), take log-softmax, gather the true-token log-prob, and mask out padding:

shift_logits = logits[:, :-1, :]
shift_labels = input_ids[:, 1:]
logprobs = F.log_softmax(shift_logits, dim=-1)
token_logprobs = logprobs.gather(-1, shift_labels.unsqueeze(-1)).squeeze(-1)
loss = -(token_logprobs * mask.float()).sum() / mask.sum()
# equivalently: F.cross_entropy(logits.view(-1, V), targets.view(-1), ignore_index=pad)

Connections:

• Cross-entropy loss + Softmax = the standard classification pipeline
• Language model training objective is cross-entropy over the vocabulary at each token position
• Perplexity $= 2^{H(P,Q)}$ — the standard metric for language models

See also: KL Divergence, Loss Functions, Softmax