An inductive bias is an assumption baked into a model's architecture that constrains what functions it can learn. It encodes prior knowledge about the problem structure.
Examples:
| Architecture | Inductive Bias | Why it helps |
|---|---|---|
| Convolutional Neural Networks (CNN) | Spatial locality + translation invariance | Features that matter in images are local and position-independent |
| Recurrent Neural Networks (RNN) | Sequential processing, Markov-like memory | Language/time series have temporal order |
| Self-Attention (Transformer) | All positions can interact equally | No prior on locality; learns relationships from data |
| Linear Regression | Linear relationship | Simplest model, strong assumption |
Stronger bias = faster learning from less data, but wrong if the assumption doesn't hold.
The transformer's advantage: minimal inductive bias — it doesn't assume locality or sequential order. This means it needs more data but can learn more general patterns. Positional Encoding is the only nod to structure.
No free lunch theorem: no model is universally best. The right inductive bias depends on the domain. CNNs dominate images precisely because the locality assumption holds; transformers are winning everywhere because they trade data efficiency for generality.
See also: Convolutional Neural Networks, Self-Attention, Bias-Variance Tradeoff