CRO Glossary
Frequentist vs Bayesian Testing
A comparison of the two dominant statistical frameworks used to analyze A/B test results and declare winners.
Frequentist and Bayesian testing are two different philosophies for interpreting experiment data, and the choice affects how results are calculated, reported, and acted on. Frequentist methods ask, 'if there were truly no difference between variants, how likely would we be to see data this extreme?' This produces the familiar p-value and requires a pre-set sample size to avoid inflating false positives. Bayesian methods instead ask, 'given the data we've observed, what is the probability that variant B is actually better than variant A?' This produces a direct probability statement that many stakeholders find more intuitive.
Practitioners care about this distinction because it changes team behavior. Frequentist tests are vulnerable to the peeking problem if teams check results early, since the math assumes a fixed sample size decided in advance. Bayesian approaches can update continuously as data arrives, which fits faster-moving teams, but they require choosing a prior belief, which can be seen as subjective or, if chosen poorly, can bias results.
For example, a frequentist report might say 'p = 0.03, reject the null hypothesis,' while a Bayesian report on the same test might say 'there is a 92% probability that the new checkout button outperforms the old one.' Many modern testing platforms, including bandit-based tools, default to Bayesian statistics specifically because the output is easier to explain to non-technical stakeholders in a readout.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply frequentist vs bayesian testing.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.