CRO Glossary
False Discovery Rate
The expected proportion of 'winning' results that are actually false positives, especially relevant when running many tests or metrics at once.
False discovery rate (FDR) matters once you're running many experiments or tracking many metrics within a single experiment. If you test 20 metrics and use a 5% significance threshold on each, you'd expect roughly one metric to show a "significant" result purely by chance even if nothing actually changed. FDR quantifies and controls for this multiple-comparisons problem, rather than trusting each p-value in isolation.
This is directly relevant to CRO teams that track a long list of secondary and guardrail metrics alongside a primary metric. Without correction, teams can convince themselves a redesign "significantly improved" some obscure metric like time-on-page-for-mobile-Safari-users, when it's really noise. Methods like the Benjamini-Hochberg procedure adjust significance thresholds to keep the expected rate of false discoveries under control across all the comparisons being made.
For example, a company running 50 experiments a quarter, each declaring significance at p<0.05, should expect around 2-3 "wins" to be false positives even if every single test had zero real effect. Programs that don't account for FDR tend to accumulate a graveyard of shipped changes that don't actually move the business, because a portion of their historical "wins" were statistical noise.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply false discovery rate.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.