CRO Glossary
Experiment Readiness Checklist
A pre-launch checklist confirming that an experiment's tracking, sample size, QA, and stakeholder sign-off are in place before it goes live.
An experiment readiness checklist is a gate teams run through before flipping an experiment on, meant to catch the avoidable mistakes that invalidate results after the fact — missing tracking events, an underpowered sample size, a broken variant on mobile, or no agreed-upon success metric. It exists because most experimentation failures aren't statistical; they're operational, caught too late to fix without restarting the test.
A typical checklist confirms: the hypothesis and primary metric are documented, the sample size and expected test duration have been calculated, instrumentation has been QA'd in a staging or preview environment, the randomization unit and target audience are correctly configured, guardrail metrics are defined, and someone has visually checked each variant across key devices and browsers. Many teams also require a second reviewer to sign off, similar to a code review, before launch.
For example, a team testing a new checkout flow might discover during readiness review that the analytics event for "payment submitted" fires on page load rather than on successful submission — a bug that would have silently inflated conversion counts in both arms and made the test's results meaningless. Catching this before launch, rather than during analysis, saves the cost of re-running the entire experiment.
Teams that skip this step tend to accumulate experiment debt: tests that have to be voided, re-run, or caveated heavily because something fundamental was wrong with the setup, undermining trust in the experimentation program as a whole.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment readiness checklist.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.