All terms

CRO Glossary

Experiment Contamination Rate

The share of users in a test who were exposed to the wrong variant or to both variants, undermining the validity of the comparison.

Experiment contamination rate quantifies how much cross-exposure or bucketing error occurred during a test — for instance, the percentage of control users who somehow also saw the treatment experience, or users who were reassigned between variants mid-test due to caching, cookie clearing, cross-device behavior, or bugs in the randomization logic. It's usually expressed as a percentage of total experiment traffic and tracked as a data-quality metric alongside the primary results.

A high contamination rate matters because it dilutes the measured treatment effect toward zero and can also introduce bias if the contamination isn't symmetric between arms. For example, if a caching bug causes 8% of control users on a checkout redesign test to briefly see the new flow, the observed lift will understate the true effect, and a team might wrongly kill a winning variant.

Practitioners monitor this by instrumenting variant assignment logging separately from variant rendering, then reconciling the two: any mismatch is contamination. A reasonable threshold varies by industry, but many teams treat anything above 1-2% as a signal to investigate before trusting the read-out, and above 5% as often disqualifying the test entirely, requiring a rerun with fixed bucketing logic.

Related terms

See this in the wild

ABWatcher watches how top teams apply experiment contamination rate.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.