Frequentist Statistics

Frequentist statistics is the traditional framework behind p-values and statistical significance thresholds like 0.05. It treats the true effect of a treatment as a fixed but unknown quantity, and asks: if there were truly no difference between variants, how likely would we be to see data this extreme or more extreme just by chance? This is different from Bayesian inference, which instead models a probability distribution over possible effect sizes and updates it as data arrives.

Most off-the-shelf A/B testing tools default to frequentist methods because they're well understood, standardized, and easy to compute at scale. The catch is that frequentist p-values are only valid if you decide your sample size in advance and don't repeatedly check results as data trickles in — doing so inflates the false positive rate, a mistake closely related to the peeking problem.

For example, a team running a frequentist test on a new pricing page would set a required sample size upfront, run the test to completion, and only then check whether the p-value falls below their threshold. If they instead check daily and stop the moment they see a 'significant' result, they're likely to declare false wins far more often than their stated 5% error rate suggests.