All terms

CRO Glossary

Experiment Validity Threats

Factors that undermine confidence that an observed effect in a test is real, causal, and generalizable.

Validity threats are the catalog of reasons a test result might be misleading even when the statistics look clean. They're typically split into internal validity (is the effect actually caused by the variant, or by something else happening at the same time?) and external validity (will the effect hold up outside this specific test, audience, or time period?). Practitioners care about this because a statistically significant result can still be wrong in practice if a validity threat went unchecked.

          Common internal threats include instrumentation changes mid-test, seasonal events coinciding with the test window, or a competitor launching a sale that shifts user behavior for both control and variant unevenly. External threats include running a test only during a holiday period and assuming the lift will hold year-round, or testing on paid traffic and assuming organic users will behave the same way.

          A concrete example: a team runs a pricing test during a month when a major holiday sale is live sitewide. The variant wins, but the lift evaporates post-launch because the effect was really driven by the sale, not the pricing change — a validity threat that a p-value alone would never reveal. Good experiment design documents explicitly list anticipated validity threats so reviewers can sanity-check results before rolling out.

Related terms

See this in the wild

ABWatcher watches how top teams apply experiment validity threats.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.