All terms

CRO Glossary

Frequentist vs Bayesian Decision Rule

The chosen framework and threshold logic a team uses to decide when test results justify shipping a variant.

A decision rule is the pre-agreed logic that turns raw test output into a ship/don't-ship call. It's separate from the underlying statistical framework itself — it's the operational policy layered on top. Two teams could both use frequentist p-values but have different decision rules: one ships at p < 0.05 regardless of effect size, another requires both significance and a minimum lift threshold before rolling out.

Without an explicit decision rule, teams tend to rationalize results after the fact — shipping flat winners because they 'feel right' or killing statistically solid results because the lift seems small. This is where a lot of organizational disagreement about experimentation comes from: not disagreement about the math, but disagreement about what the math should be used to justify.

A good decision rule is written down before the test launches, as part of the experiment design document, and covers three things: what threshold counts as a win, what happens with an inconclusive result, and who has authority to override the rule. For example: 'Ship if MDE is met AND guardrail metrics are flat or improved AND the result holds through the pre-registered stopping point.'

Teams that skip this step often end up with inconsistent shipping standards across experimenters, which erodes trust in the whole program over time — every 'win' gets second-guessed because nobody agrees on what a win actually required.

Related terms

See this in the wild

ABWatcher watches how top teams apply frequentist vs bayesian decision rule.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.