Hypothesis Testing

A hypothesis test asks whether observed data is surprising under a null hypothesis.

Null hypothesis $H_0$: default claim, often "no effect" or "no difference."

Alternative hypothesis $H_1$: claim you need evidence for.

Example:

• $H_0$: model A and model B have equal expected accuracy
• $H_1$: model A has higher expected accuracy than model B

Test statistic: a number computed from the data whose distribution is known or approximated under $H_0$.

p-value: probability, assuming $H_0$ is true, of observing a test statistic at least as extreme as the one observed.

Decision rule: reject $H_0$ if $p < \alpha$, where $\alpha$ is the significance level, often 0.05.

What a p-value is not:

• It is not $P(H_0 \mid \text{data})$
• It is not the probability the result is false
• It does not measure effect size
• It does not guarantee practical importance

In ML: hypothesis tests can compare models, but they must respect dependence. If two models are evaluated on the same test examples, use paired tests or bootstrap differences, not independent-sample tests.

See also: Statistical Power, Statistical Significance vs Practical Significance, Bootstrap and Resampling