Cross-Validation

Cross-validation provides honest estimates of model performance on unseen data.

K-Fold CV:

1. Split data into $k$ equal folds (typically $k = 5$ or $10$)
2. For each fold $i$: train on all folds except $i$, evaluate on fold $i$
3. Average performance across all $k$ evaluations

Why not just train/test split?

• Single split is high variance — performance depends on which data ended up in test
• K-fold uses all data for both training and evaluation
• Gives mean and standard deviation of performance

Variants:

• Stratified K-fold — preserves class distribution in each fold (use for imbalanced data)
• Leave-one-out (LOO) — $k = n$, most expensive but lowest bias
• Repeated K-fold — run K-fold multiple times with different random splits

Use cases:

• Model selection — compare models (e.g., different hyperparameters) by their CV score
• Hyperparameter tuning — grid/random search over hyperparameters, using CV to evaluate each
• Final evaluation — estimate generalization error of the chosen model

Pitfall: never tune hyperparameters on the test set. Use nested CV or a separate validation set for tuning, then evaluate once on test.

See also: Bias-Variance Tradeoff, Evaluation Metrics