Multimodality

Multimodal models extend a language model to consume (and sometimes produce) images, audio, or video by mapping each modality into the same embedding space the LLM already operates in.

Vision Transformer (ViT): split an image into fixed-size patches, linearly project each patch into a vector, prepend a CLS token, add position embeddings, and run a standard transformer encoder — treating patches exactly like tokens.

LLaVA bridges a frozen vision encoder to an LLM with a single learned projection ($D = 4096$ for the LLM, 1024 for CLIP):

• image $[224, 224, 3]$ → patchify
• patches $[256, 588]$ → linear projection (applied per patch)
• patch embeddings $[256, 1024]$ → prepend CLS, add position embeddings
• sequence $[257, 1024]$ → CLIP encoder (24 layers)
• CLIP output $[256, 1024]$ → projection $W$ (per token)
• visual embeddings $[256, 4096]$ → concatenate with text embeddings $[N, 4096]$
• full sequence $[256 + N, 4096]$ → LLM transformer

Handling arbitrary resolution:

• LLaVA-NeXT — split a high-res image into multiple crops, encode each, concatenate
• Qwen2-VL — replace learned absolute position embeddings (which fixed the image size) with 2D-RoPE, encoding position as (row, column) coordinates generated on the fly → generalizes to any resolution and aspect ratio

CLIP is trained with contrastive learning: maximize the similarity of text/image embeddings for correct pairings and minimize it for mismatches. This shared space is what lets a projection align vision features with language.

Current paradigm: understanding uses a ViT encoder → features; generation uses diffusion operating in pixel space.

See also: Self-Attention, Positional Encoding, Embeddings, Transformers and LLMs