Pretraining

Pretraining teaches an LLM to predict the next token on massive text corpora — the foundational stage of the LLM pipeline.

Objective: given tokens $t_1, \dots, t_n$, minimize:

This is cross-entropy loss on next-token prediction (cross-entropy = negative log-likelihood).

What the model learns:

• Grammar, syntax, factual knowledge, reasoning patterns, coding ability
• All from the single objective of predicting what comes next
• The training data is the entire internet (+ books, code, etc.)

Scale:

• Hundreds of billions to trillions of tokens
• Models from 1B to 1T+ parameters
• Training runs cost $1M–$100M+
• Scaling Laws: performance improves predictably with more data, model size, and compute

What pretraining does NOT give you:

• Instruction following (the model continues text, doesn't answer questions)
• Safety/alignment
• These come from Supervised Fine-Tuning and RLHF/DPO in later stages

The full pipeline: Pretraining → SFT → RLHF or DPO

See also: Tokenization, Scaling Laws, Autoregressive Generation