CRO Glossary
Sample Pollution
Contamination of an experiment's data by users, bots, or traffic that shouldn't have been included in the analysis.
Sample pollution happens when the population actually measured in a test diverges from the population you intended to measure. Common culprits include internal employees and QA traffic browsing the live site, bots and scrapers hitting variant pages, users who cleared cookies and got re-bucketed into a different arm, or third-party tools caching a stale variant and serving it inconsistently. Unlike sample ratio mismatch, which is a symptom you can detect from allocation counts, sample pollution can look statistically normal while still quietly biasing the results.
Practitioners care because polluted samples produce results that don't generalize — a lift measured on 8% bot traffic mixed with real users tells you nothing reliable about real user behavior, yet the dashboard will still report a confident-looking p-value. It's especially dangerous because it's invisible unless you specifically audit for it: traffic source, user agent, device fingerprint, or internal IP ranges.
A concrete example: an e-commerce team runs a checkout test and sees a huge conversion lift on the challenger. Later they discover a marketing team accidentally drove a paid traffic surge with a coupon code that landed disproportionately in the challenger arm due to a targeting misconfiguration. The 'winning' variant was really just measuring higher-intent, discount-seeking shoppers, not a genuine product improvement.
Mitigation is mostly preventive: filter known bots and internal IPs before analysis, exclude QA and staging sessions, monitor for anomalous traffic source spikes, and re-check randomization health (like SRM) whenever a result looks too good to be true.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply sample pollution.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.