CRO Glossary
Metric Dilution
When a test's effect is real but gets averaged away because it only affects a small slice of the measured population.
Metric dilution happens when you measure an outcome across a broad population, but your change only meaningfully affects a narrow subset of that population. The average effect gets watered down by all the unaffected users, making a genuinely strong result look flat or negligible in the top-line number.
This is a common trap in CRO because primary metrics are often chosen for business relevance (like overall conversion rate) rather than sensitivity to the specific change being tested. A redesign of the checkout page for mobile Safari users, for example, might produce a huge lift for that segment but barely move the needle on 'site-wide conversion rate' if mobile Safari is only 8% of traffic.
The practical fix is to pre-specify a segment-level or mechanism-level metric alongside the top-line one — essentially deciding in advance where you expect the effect to show up, and checking there directly rather than relying solely on the diluted aggregate. This is different from post-hoc segment slicing done after seeing a null result, which invites false positives; the key is committing to the affected-segment metric before the test runs.
Teams that ignore dilution risk shutting down real wins because the aggregate metric wasn't sensitive enough to detect them, especially in tests aimed at a specific device, geography, or user lifecycle stage.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply metric dilution.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.