Experiment Contamination

Experiment contamination happens when users assigned to one arm of a test are exposed, directly or indirectly, to the other arm — making the two groups less different than intended and biasing the measured effect toward zero (or in unpredictable directions). It's one of the most common and hardest-to-detect ways an experiment quietly fails without ever throwing an obvious error like SRM.

Contamination shows up in several forms: a user logs in on a different device and gets bucketed into a different variant than on their first device; a caching layer serves the wrong variant to a subset of "control" users; word-of-mouth or screenshots let people manually navigate to a feature meant only for the treatment group; or a bug lets both variants render on the same page simultaneously. In server-side and client-side testing alike, anything that breaks the consistency of the randomization unit's assignment across sessions is a contamination risk.

A classic example is testing a new pricing page with a randomization unit of "session" instead of "user" — the same customer might see the control price on their laptop and the treatment price on their phone, then compare notes or get confused, undermining both the data and the user experience.

Because contamination usually shrinks observed effects rather than eliminating them outright, it's dangerous: a genuinely effective change can look like a null result. Guarding against it requires consistent assignment keys, QA across devices and sessions, and monitoring for unexpected cross-exposure before trusting a flat or muted result.