Loss Functions

The loss function measures how wrong the model's predictions are. Training minimizes it via Backpropagation.

Regression:

Mean Squared Error (MSE): $L = \frac{1}{n}\sum_i(y_i - \hat{y}_i)^2$

• Equivalent to MLE assuming Gaussian noise
• Penalizes large errors quadratically → sensitive to outliers
• Gradient: $\frac{\partial L}{\partial \hat{y}_i} = \frac{2}{n}(\hat{y}_i - y_i)$

Classification:

Binary cross-entropy: $L = -\frac{1}{n}\sum_i [y_i\log\hat{y}_i + (1-y_i)\log(1-\hat{y}_i)]$

• Equivalent to negative log-likelihood of Bernoulli distribution
• Used with sigmoid output

Categorical cross-entropy: $L = -\frac{1}{n}\sum_i\sum_k y_{ik}\log\hat{y}_{ik}$

• Equivalent to negative log-likelihood of categorical distribution
• Used with softmax output
• One-hot $y$: simplifies to $L = -\frac{1}{n}\sum_i \log\hat{y}_{i,c_i}$ where $c_i$ is the true class

Connection: minimizing cross-entropy = minimizing KL Divergence from the true distribution (since $H(p,q) = H(p) + D_\text{KL}(p \| q)$ and $H(p)$ is constant).

In LLM pretraining: cross-entropy on next-token prediction is the standard objective → Pretraining.

See also: Maximum Likelihood Estimation, Logistic Regression