CRO Glossary
Frequentist Statistics
A statistical approach that judges results by how likely the observed data would be under a fixed null hypothesis.
Frequentist statistics is the traditional framework behind p-values and statistical significance thresholds like 0.05. It treats the true effect of a treatment as a fixed but unknown quantity, and asks: if there were truly no difference between variants, how likely would we be to see data this extreme or more extreme just by chance? This is different from Bayesian inference, which instead models a probability distribution over possible effect sizes and updates it as data arrives.
Most off-the-shelf A/B testing tools default to frequentist methods because they're well understood, standardized, and easy to compute at scale. The catch is that frequentist p-values are only valid if you decide your sample size in advance and don't repeatedly check results as data trickles in — doing so inflates the false positive rate, a mistake closely related to the peeking problem.
For example, a team running a frequentist test on a new pricing page would set a required sample size upfront, run the test to completion, and only then check whether the p-value falls below their threshold. If they instead check daily and stop the moment they see a 'significant' result, they're likely to declare false wins far more often than their stated 5% error rate suggests.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply frequentist statistics.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.