All terms

CRO Glossary

Variance Reduction

Statistical techniques that reduce noise in experiment metrics so true effects are detected faster with smaller samples.

Variance reduction refers to a family of statistical methods used to strip unwanted noise out of an experiment's outcome metric before comparing groups. The core idea is that a lot of the variability in metrics like revenue per user or conversion rate has nothing to do with the treatment — it comes from pre-existing differences between users, seasonality, or random behavioral noise. By modeling and removing that predictable variance, you can see the treatment effect more clearly without needing to collect more traffic.

The most common technique is CUPED (Controlled-experiment Using Pre-Existing Data), which adjusts each user's outcome using their pre-experiment behavior as a covariate. Other approaches include stratified sampling, regression adjustment, and using log-transforms or winsorized metrics to tame skew. These methods don't change the actual effect size — they change how precisely you can measure it, which shrinks confidence intervals and shortens the time needed to reach a reliable read.

Practitioners care about this because traffic is often the binding constraint on experimentation velocity. A team that can cut variance by 30% effectively gets the equivalent of running with much more traffic, without waiting longer or diverting more users into a test. It's especially valuable for high-variance metrics like revenue, where a handful of whale customers can swamp any signal from the treatment itself.

For example, an e-commerce team testing a new checkout flow might use each user's average order value over the prior 30 days as a covariate. Two users who see the same treatment but historically spend very differently get 'corrected' toward a comparable baseline, so the remaining variance is driven mostly by the change being tested rather than by who happened to land in which bucket.

Related terms

See this in the wild

ABWatcher watches how top teams apply variance reduction.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.