CRO Glossary
Experiment Validity Threats
Factors that undermine confidence that an observed effect in a test is real, causal, and generalizable.
Validity threats are the catalog of reasons a test result might be misleading even when the statistics look clean. They're typically split into internal validity (is the effect actually caused by the variant, or by something else happening at the same time?) and external validity (will the effect hold up outside this specific test, audience, or time period?). Practitioners care about this because a statistically significant result can still be wrong in practice if a validity threat went unchecked.
Common internal threats include instrumentation changes mid-test, seasonal events coinciding with the test window, or a competitor launching a sale that shifts user behavior for both control and variant unevenly. External threats include running a test only during a holiday period and assuming the lift will hold year-round, or testing on paid traffic and assuming organic users will behave the same way.
A concrete example: a team runs a pricing test during a month when a major holiday sale is live sitewide. The variant wins, but the lift evaporates post-launch because the effect was really driven by the sale, not the pricing change — a validity threat that a p-value alone would never reveal. Good experiment design documents explicitly list anticipated validity threats so reviewers can sanity-check results before rolling out.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment validity threats.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.