Statistical Power

Statistical power is the probability that a test correctly detects a real effect.

\text{Power} = P(\text{reject } H_0 \mid H_1 \text{ is true})

Errors in hypothesis testing:

Decision	Reality	Name
Reject $H_0$	$H_0$ true	Type I error / false positive
Fail to reject $H_0$	$H_1$ true	Type II error / false negative

The Type I error rate is controlled by $\alpha$ :

P(\text{Type I error}) = \alpha

The Type II error rate is $\beta$ , so:

\text{Power} = 1 - \beta

Power increases when:

Sample size increases
Effect size increases
Measurement noise decreases
Significance threshold $\alpha$ becomes less strict

Low power is dangerous: a non-significant result may mean "not enough data," not "no effect."

In ML: small benchmark differences often require large test sets to detect reliably. If the test set is small, "model A beats model B by 0.3%" may be noise.

Statistical Power

Linked from