CRO Glossary
Outlier Trimming
The practice of removing or capping extreme data points before analyzing an experiment to prevent them from distorting results.
Outlier trimming is a data-cleaning step applied before or during experiment analysis, where unusually extreme values — a single customer placing a $50,000 order, a bot generating thousands of pageviews, or a session lasting 12 hours due to an idle tab — are removed, capped, or winsorized so they don't dominate the metric being measured.
This matters because many CRO metrics, especially revenue-per-visitor or average order value, are highly sensitive to a small number of extreme values. A single whale purchase in a small sample can flip which variant looks like the 'winner,' even though the difference has nothing to do with the design change being tested. Without trimming, a test's outcome can hinge on one outlier rather than a genuine shift in user behavior.
A typical approach is to cap revenue values at a chosen percentile (e.g., the 99th) or exclude sessions above a defined threshold, then rerun the analysis to see if the result holds. Teams often report both the trimmed and untrimmed results to demonstrate the effect isn't an artifact of extreme values.
The main risk is applying trimming rules inconsistently or after seeing results, which opens the door to cherry-picking a threshold that produces the desired outcome — a form of p-hacking. Best practice is to define outlier-handling rules in the pre-registration or analysis plan, before the experiment runs, so trimming decisions aren't influenced by the data itself.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply outlier trimming.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.