CRO Glossary
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Pre-test analysis is the diagnostic work a team does before an experiment goes live: checking that tracking fires correctly, that the baseline conversion rate is stable over time, that traffic volume actually supports the planned sample size, and that no seasonal or external event will contaminate the read. It's the difference between designing a test in a spreadsheet and knowing the test will actually produce a trustworthy answer.
Practitioners care about this because most "bad test" post-mortems trace back to something that was knowable in advance — a metric that was too noisy to move, an audience segment that was too small, or an instrumentation bug that silently undercounted conversions. Pre-test analysis catches these before they waste weeks of traffic and stakeholder patience.
A typical pre-test checklist includes pulling 4-8 weeks of historical data to check for trend or seasonality, confirming the metric's variance to sanity-check the MDE, running a dry-run or QA pass on the tracking implementation, and confirming that the randomization unit and eligible audience match what the hypothesis assumes. For example, before testing a new checkout flow, a team might discover in pre-test analysis that 15% of sessions never fire the checkout-start event on mobile Safari — fixing that first saves the whole experiment from being unusable.
Skipping this step is a common root cause of SRM, underpowered tests, and misleading read-outs that only surface after the test has already run its course.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply pre-test analysis.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.