CRO Glossary
Experiment Contamination Rate
The share of users in a test who were exposed to the wrong variant or to both variants, undermining the validity of the comparison.
Experiment contamination rate quantifies how much cross-exposure or bucketing error occurred during a test — for instance, the percentage of control users who somehow also saw the treatment experience, or users who were reassigned between variants mid-test due to caching, cookie clearing, cross-device behavior, or bugs in the randomization logic. It's usually expressed as a percentage of total experiment traffic and tracked as a data-quality metric alongside the primary results.
A high contamination rate matters because it dilutes the measured treatment effect toward zero and can also introduce bias if the contamination isn't symmetric between arms. For example, if a caching bug causes 8% of control users on a checkout redesign test to briefly see the new flow, the observed lift will understate the true effect, and a team might wrongly kill a winning variant.
Practitioners monitor this by instrumenting variant assignment logging separately from variant rendering, then reconciling the two: any mismatch is contamination. A reasonable threshold varies by industry, but many teams treat anything above 1-2% as a signal to investigate before trusting the read-out, and above 5% as often disqualifying the test entirely, requiring a rerun with fixed bucketing logic.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment contamination rate.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.