Bootstrap and Resampling

The bootstrap estimates uncertainty by resampling from the observed data.

Given data $x_1, \dots, x_n$:

1. Sample $n$ observations with replacement from the dataset
2. Compute the statistic of interest
3. Repeat many times
4. Use the empirical distribution of bootstrap statistics to estimate uncertainty

This approximates the sampling distribution of the statistic.

Why sample with replacement? it mimics drawing new datasets from the empirical distribution.

Uses:

• Standard errors
• Confidence intervals
• Comparing models
• Estimating uncertainty for statistics without simple formulas

Bootstrap confidence interval: take percentiles of the bootstrap distribution, such as the 2.5th and 97.5th percentiles for a 95% interval.

Limitations:

• Assumes the sample represents the population
• Can fail with very small samples
• Can fail for extreme-value statistics
• Must preserve dependence structure, such as paired predictions or time series blocks

In ML: to compare two models on the same test set, bootstrap the per-example metric differences. This keeps the paired structure intact.

See also: Sampling Distributions, Confidence Intervals, Evaluation Metrics