CRO Glossary
Test Vetoing
The practice of stopping or blocking an experiment from launching based on subjective judgment rather than pre-agreed statistical or business criteria.
Test vetoing happens when a stakeholder — often a senior designer, executive, or product owner — kills or overrides an experiment because they personally dislike the result, distrust the data, or simply prefer the original design. It's distinct from a legitimate kill decision based on pre-registered kill criteria or a guardrail metric breach; vetoing is driven by opinion after the fact, which undermines the point of testing in the first place.
This matters because a culture that allows vetoing quietly converts an experimentation program into a confirmation exercise: teams only "listen to the data" when it agrees with what leadership already wanted. Over time this erodes trust in the program, discourages analysts from reporting inconvenient results, and biases the historical record of what "worked," making future prioritization decisions unreliable.
For example, a checkout redesign might lift conversion by 4% with strong significance, but a VP vetoes the rollout because they feel the new layout looks "unpolished," and the change is never shipped. A healthier alternative is to define kill criteria and decision thresholds before the test launches, so any override has to be justified against those pre-agreed rules rather than post-hoc taste.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply test vetoing.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.