CRO Glossary
Outlier Trimming
The practice of removing or capping extreme data points before analyzing an experiment to prevent them from distorting results.
Outlier trimming is a data-cleaning step applied before or during experiment analysis, where unusually extreme values — a single customer placing a $50,000 order, a bot generating thousands of pageviews, or a session lasting 12 hours due to an idle tab — are removed, capped, or winsorized so they don't dominate the metric being measured.
This matters because many CRO metrics, especially revenue-per-visitor or average order value, are highly sensitive to a small number of extreme values. A single whale purchase in a small sample can flip which variant looks like the 'winner,' even though the difference has nothing to do with the design change being tested. Without trimming, a test's outcome can hinge on one outlier rather than a genuine shift in user behavior.
A typical approach is to cap revenue values at a chosen percentile (e.g., the 99th) or exclude sessions above a defined threshold, then rerun the analysis to see if the result holds. Teams often report both the trimmed and untrimmed results to demonstrate the effect isn't an artifact of extreme values.
The main risk is applying trimming rules inconsistently or after seeing results, which opens the door to cherry-picking a threshold that produces the desired outcome — a form of p-hacking. Best practice is to define outlier-handling rules in the pre-registration or analysis plan, before the experiment runs, so trimming decisions aren't influenced by the data itself.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply outlier trimming.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.