CRO Glossary
Confidence Washing
Presenting an underpowered or flawed test result with false statistical authority to justify a decision already made.
Confidence washing happens when a team wraps a weak or inconclusive experiment in the language of rigor — p-values, confidence intervals, dashboards — to make a predetermined decision look data-driven. The numbers are real, but the process behind them was bent to fit a conclusion: the test was stopped early because it looked good, a segment was cherry-picked because it hit significance, or a low-powered test was reported as a clear win because stakeholders wanted to ship.
This matters because it erodes trust in the whole experimentation program. Once a team discovers that a 'statistically significant' result was actually the product of peeking, selective segment reporting, or a suspiciously convenient stopping rule, they start questioning every past result, not just the flawed one. It's a cultural problem as much as a statistical one — usually driven by pressure to ship, not by dishonesty.
A typical example: a redesign test runs for three days, shows a positive trend in one browser segment, and gets reported to leadership as 'the new checkout increases conversion by 12%, with statistical significance,' omitting that the overall result was flat and the sample size was a fraction of what the pre-registered plan called for.
The fix is procedural: pre-register hypotheses and stopping rules, report overall results alongside any segment cuts, and require analysts to flag when a result technically clears a significance threshold but fails other quality checks (power, duration, SRM).
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply confidence washing.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.