CRO Glossary
Effect Size
The magnitude of the difference between a variant and control, expressed in absolute or relative terms.
Effect size answers 'how big was the change,' as opposed to statistical significance, which answers 'how confident are we that the change is real.' A test can be statistically significant with a tiny effect size, or show a huge effect size that isn't statistically significant because the sample was too small. Practitioners need both numbers to make good decisions.
Effect size is usually reported as an absolute difference (control converts at 4.0%, variant at 4.6%, a 0.6 percentage point lift) or a relative difference (a 15% relative lift). Relative lift is often more useful for comparing tests across different baseline conversion rates, while absolute lift is more useful for estimating real business impact like additional revenue.
Effect size is also the key input to a minimum detectable effect calculation before a test starts: you decide the smallest effect size that would be worth detecting, and that determines the sample size you need. A test that later observes an effect size below that threshold, even if directionally positive, generally shouldn't be treated as a meaningful win.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply effect size.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.