CRO Glossary
Variance Reduction
Statistical techniques that reduce noise in experiment metrics so true effects are detected faster with smaller samples.
Variance reduction refers to a family of statistical methods used to strip unwanted noise out of an experiment's outcome metric before comparing groups. The core idea is that a lot of the variability in metrics like revenue per user or conversion rate has nothing to do with the treatment — it comes from pre-existing differences between users, seasonality, or random behavioral noise. By modeling and removing that predictable variance, you can see the treatment effect more clearly without needing to collect more traffic.
The most common technique is CUPED (Controlled-experiment Using Pre-Existing Data), which adjusts each user's outcome using their pre-experiment behavior as a covariate. Other approaches include stratified sampling, regression adjustment, and using log-transforms or winsorized metrics to tame skew. These methods don't change the actual effect size — they change how precisely you can measure it, which shrinks confidence intervals and shortens the time needed to reach a reliable read.
Practitioners care about this because traffic is often the binding constraint on experimentation velocity. A team that can cut variance by 30% effectively gets the equivalent of running with much more traffic, without waiting longer or diverting more users into a test. It's especially valuable for high-variance metrics like revenue, where a handful of whale customers can swamp any signal from the treatment itself.
For example, an e-commerce team testing a new checkout flow might use each user's average order value over the prior 30 days as a covariate. Two users who see the same treatment but historically spend very differently get 'corrected' toward a comparable baseline, so the remaining variance is driven mostly by the change being tested rather than by who happened to land in which bucket.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply variance reduction.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.