CRO Glossary
False Discovery Rate
The expected proportion of 'winning' results that are actually false positives, especially relevant when running many tests or metrics at once.
False discovery rate (FDR) matters once you're running many experiments or tracking many metrics within a single experiment. If you test 20 metrics and use a 5% significance threshold on each, you'd expect roughly one metric to show a "significant" result purely by chance even if nothing actually changed. FDR quantifies and controls for this multiple-comparisons problem, rather than trusting each p-value in isolation.
This is directly relevant to CRO teams that track a long list of secondary and guardrail metrics alongside a primary metric. Without correction, teams can convince themselves a redesign "significantly improved" some obscure metric like time-on-page-for-mobile-Safari-users, when it's really noise. Methods like the Benjamini-Hochberg procedure adjust significance thresholds to keep the expected rate of false discoveries under control across all the comparisons being made.
For example, a company running 50 experiments a quarter, each declaring significance at p<0.05, should expect around 2-3 "wins" to be false positives even if every single test had zero real effect. Programs that don't account for FDR tend to accumulate a graveyard of shipped changes that don't actually move the business, because a portion of their historical "wins" were statistical noise.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply false discovery rate.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.