CRO Glossary
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
Experiment contamination happens when users assigned to one arm of a test are exposed, directly or indirectly, to the other arm — making the two groups less different than intended and biasing the measured effect toward zero (or in unpredictable directions). It's one of the most common and hardest-to-detect ways an experiment quietly fails without ever throwing an obvious error like SRM.
Contamination shows up in several forms: a user logs in on a different device and gets bucketed into a different variant than on their first device; a caching layer serves the wrong variant to a subset of "control" users; word-of-mouth or screenshots let people manually navigate to a feature meant only for the treatment group; or a bug lets both variants render on the same page simultaneously. In server-side and client-side testing alike, anything that breaks the consistency of the randomization unit's assignment across sessions is a contamination risk.
A classic example is testing a new pricing page with a randomization unit of "session" instead of "user" — the same customer might see the control price on their laptop and the treatment price on their phone, then compare notes or get confused, undermining both the data and the user experience.
Because contamination usually shrinks observed effects rather than eliminating them outright, it's dangerous: a genuinely effective change can look like a null result. Guarding against it requires consistent assignment keys, QA across devices and sessions, and monitoring for unexpected cross-exposure before trusting a flat or muted result.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment contamination.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.