CRO Glossary
Statistical Significance
A measure of how unlikely it is that an observed result happened by random chance alone, typically expressed as a p-value.
Statistical significance answers a narrow question: if there were truly no difference between variants, how surprising would this data be? It's usually summarized as a p-value, and a common threshold is p < 0.05, meaning there's less than a 5% chance of seeing a difference this large (or larger) if the variants were actually identical.
Significance is not the same as importance. A test can be statistically significant but practically trivial (a 0.1% lift on a huge sample), or it can be a large, meaningful effect that isn't yet significant because the sample is too small — often because the test wasn't given enough time to reach its planned minimum detectable effect. It also says nothing about the probability that the effect is real in a Bayesian sense, which is a common point of confusion versus Bayesian inference.
A frequent mistake is checking significance repeatedly during a test and stopping as soon as it crosses the threshold — this is the peeking problem, and it inflates false positives well above the stated 5%. Proper use of significance requires deciding your sample size and test duration in advance, or using a sequential testing method designed to allow continuous monitoring.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply statistical significance.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.