Embeddings

An embedding maps discrete objects (words, tokens, users, items) into a continuous vector space $\mathbb{R}^d$ where geometric relationships encode semantic relationships.

Mechanically: an embedding layer is a lookup table — a matrix $E \in \mathbb{R}^{|V| \times d}$ where row $i$ is the vector for item $i$. This is equivalent to multiplying a one-hot vector by $E$, but implemented as an index lookup for efficiency.

What makes embeddings powerful:

• Similar items end up nearby in the space (small cosine distance)
• Directions can encode relationships: the classic $\text{king} - \text{man} + \text{woman} \approx \text{queen}$
• Dimensionality reduction: vocabulary of 50k tokens → 768-dimensional vectors (orders of magnitude compression)
• Learned, not hand-crafted — the representation is optimized end-to-end for the task

Key examples:

• Word2Vec / GloVe — static word embeddings trained on co-occurrence
• Token embeddings in transformers — the first layer of GPT, BERT, etc.
• Positional Encoding — added to token embeddings to inject sequence order
• Recommendation systems — user and item embeddings; predict preference via Dot Product
• Retrieval — embed queries and documents, find nearest neighbors via cosine similarity

The embedding space is the representation. Most of deep learning is about learning good embeddings — the final classifier or prediction head is often trivially simple (a linear layer) on top of a rich embedding.

See also: Positional Encoding, Norms and Distance Metrics, Tokenization