All terms

CRO Glossary

Sample Pollution

Contamination of an experiment's data by users, bots, or traffic that shouldn't have been included in the analysis.

Sample pollution happens when the population actually measured in a test diverges from the population you intended to measure. Common culprits include internal employees and QA traffic browsing the live site, bots and scrapers hitting variant pages, users who cleared cookies and got re-bucketed into a different arm, or third-party tools caching a stale variant and serving it inconsistently. Unlike sample ratio mismatch, which is a symptom you can detect from allocation counts, sample pollution can look statistically normal while still quietly biasing the results.

Practitioners care because polluted samples produce results that don't generalize — a lift measured on 8% bot traffic mixed with real users tells you nothing reliable about real user behavior, yet the dashboard will still report a confident-looking p-value. It's especially dangerous because it's invisible unless you specifically audit for it: traffic source, user agent, device fingerprint, or internal IP ranges.

A concrete example: an e-commerce team runs a checkout test and sees a huge conversion lift on the challenger. Later they discover a marketing team accidentally drove a paid traffic surge with a coupon code that landed disproportionately in the challenger arm due to a targeting misconfiguration. The 'winning' variant was really just measuring higher-intent, discount-seeking shoppers, not a genuine product improvement.

Mitigation is mostly preventive: filter known bots and internal IPs before analysis, exclude QA and staging sessions, monitor for anomalous traffic source spikes, and re-check randomization health (like SRM) whenever a result looks too good to be true.

Related terms

See this in the wild

ABWatcher watches how top teams apply sample pollution.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.