CRO Glossary
Metric Dilution
When a test's effect is real but gets averaged away because it only affects a small slice of the measured population.
Metric dilution happens when you measure an outcome across a broad population, but your change only meaningfully affects a narrow subset of that population. The average effect gets watered down by all the unaffected users, making a genuinely strong result look flat or negligible in the top-line number.
This is a common trap in CRO because primary metrics are often chosen for business relevance (like overall conversion rate) rather than sensitivity to the specific change being tested. A redesign of the checkout page for mobile Safari users, for example, might produce a huge lift for that segment but barely move the needle on 'site-wide conversion rate' if mobile Safari is only 8% of traffic.
The practical fix is to pre-specify a segment-level or mechanism-level metric alongside the top-line one — essentially deciding in advance where you expect the effect to show up, and checking there directly rather than relying solely on the diluted aggregate. This is different from post-hoc segment slicing done after seeing a null result, which invites false positives; the key is committing to the affected-segment metric before the test runs.
Teams that ignore dilution risk shutting down real wins because the aggregate metric wasn't sensitive enough to detect them, especially in tests aimed at a specific device, geography, or user lifecycle stage.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply metric dilution.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.