CRO Glossary
Statistical Significance
A measure of how unlikely it is that an observed result happened by random chance alone, typically expressed as a p-value.
Statistical significance answers a narrow question: if there were truly no difference between variants, how surprising would this data be? It's usually summarized as a p-value, and a common threshold is p < 0.05, meaning there's less than a 5% chance of seeing a difference this large (or larger) if the variants were actually identical.
Significance is not the same as importance. A test can be statistically significant but practically trivial (a 0.1% lift on a huge sample), or it can be a large, meaningful effect that isn't yet significant because the sample is too small — often because the test wasn't given enough time to reach its planned minimum detectable effect. It also says nothing about the probability that the effect is real in a Bayesian sense, which is a common point of confusion versus Bayesian inference.
A frequent mistake is checking significance repeatedly during a test and stopping as soon as it crosses the threshold — this is the peeking problem, and it inflates false positives well above the stated 5%. Proper use of significance requires deciding your sample size and test duration in advance, or using a sequential testing method designed to allow continuous monitoring.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply statistical significance.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.