CRO Glossary
Experiment Governance
The set of policies, roles, and review processes that determine how experiments are approved, run, and interpreted across an organization.
Experiment governance is the operating model that sits above any individual test — it answers questions like who can launch an experiment, what statistical standards are mandatory, which metrics require guardrail sign-off, and how conflicting or overlapping tests get resolved. As experimentation programs scale from a handful of tests a quarter to dozens running concurrently across teams, informal governance breaks down fast: two teams end up testing on the same page without knowing it, or someone ships a 'winner' that was never checked for sample ratio mismatch.
This matters because bad governance quietly destroys the credibility of good statistics. You can have a rigorous stats team and still get garbage decisions if there's no enforced process for pre-registration, no shared calendar of live tests, and no consistent bar for what counts as a guardrail breach. Governance is the layer that makes rigor actually stick at an organizational level rather than living only in one analyst's head.
In practice, governance often shows up as a lightweight review board or checklist: every experiment needs an experiment design document, a named guardrail metric, a pre-registered stopping rule, and sign-off from someone outside the requesting team before it launches. A concrete example is a mid-size company that instituted a weekly 15-minute experiment review where every new test gets logged against a shared roadmap; within a quarter, test collisions and duplicate work on the same page dropped to near zero, and post-hoc disputes about whether a result 'really' won became rare because the rules were agreed in advance.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment governance.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.