CRO Glossary
Regression to the Mean
The tendency for extreme early results in a test to move closer to average as more data comes in.
Regression to the mean describes why a variant that looks like a huge winner (or loser) in the first day or two of a test often becomes much less dramatic — or disappears entirely — as the sample grows. Early results are based on small samples and are disproportionately influenced by a few unusual visitors or random noise.
This is a major reason the peeking problem is dangerous: stopping a test early because you see a stunning 40% lift often means you've caught a temporary extreme that will regress toward a much smaller, or nonexistent, effect once more data arrives.
A good practical rule: treat any early result, especially a dramatically positive one, with suspicion until the test reaches its pre-calculated sample size based on the minimum detectable effect (MDE). If a result still holds at full sample size, it's far more trustworthy than a similar-looking result seen on day one.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply regression to the mean.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.