CRO Glossary
Frequentist vs Bayesian Decision Rule
The chosen framework and threshold logic a team uses to decide when test results justify shipping a variant.
A decision rule is the pre-agreed logic that turns raw test output into a ship/don't-ship call. It's separate from the underlying statistical framework itself — it's the operational policy layered on top. Two teams could both use frequentist p-values but have different decision rules: one ships at p < 0.05 regardless of effect size, another requires both significance and a minimum lift threshold before rolling out.
Without an explicit decision rule, teams tend to rationalize results after the fact — shipping flat winners because they 'feel right' or killing statistically solid results because the lift seems small. This is where a lot of organizational disagreement about experimentation comes from: not disagreement about the math, but disagreement about what the math should be used to justify.
A good decision rule is written down before the test launches, as part of the experiment design document, and covers three things: what threshold counts as a win, what happens with an inconclusive result, and who has authority to override the rule. For example: 'Ship if MDE is met AND guardrail metrics are flat or improved AND the result holds through the pre-registered stopping point.'
Teams that skip this step often end up with inconsistent shipping standards across experimenters, which erodes trust in the whole program over time — every 'win' gets second-guessed because nobody agrees on what a win actually required.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply frequentist vs bayesian decision rule.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.