CRO Glossary
Experiment Health Check
A pre- and mid-flight audit of an experiment's setup and data quality to confirm results can be trusted.
An experiment health check is the routine of verifying that a test is actually measuring what it claims to measure before anyone looks at the results and calls a winner. It covers things like whether tracking events are firing correctly, whether traffic is splitting as configured, whether users are being bucketed consistently across sessions and devices, and whether the test is running long enough to cover normal business cycles (like a full week, including weekends).
Teams care about this because a huge share of 'surprising' experiment results turn out to be data problems rather than real user behavior. A button that lifted conversion by 40% is exciting until you discover the tracking pixel double-fired for the variant. Catching that early saves embarrassment and wasted engineering time on a fake winner.
In practice, a health check is a short checklist run 24-48 hours after launch and again before the readout: confirm sample sizes match expected allocation, check for sample ratio mismatch, spot-check a few user sessions manually, and make sure no other experiment or feature flag is silently overlapping with the same audience. Some CRO teams build this into their tooling as an automated dashboard that flags anomalies before a human ever opens the results.
Example: a team launches a checkout redesign test and, two days in, notices the mobile variant has half the expected sessions. The health check catches a JavaScript error breaking the variant on older Android browsers — before the team wastes three weeks running a broken test.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment health check.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.