CRO Glossary
P-hacking
Manipulating data, metrics, or stopping rules until a test result crosses the significance threshold, producing misleading findings.
P-hacking happens when someone runs a test, doesn't get the result they hoped for, and then keeps tweaking things — adding segments, removing outliers, extending the test, or checking dozens of secondary metrics — until something turns up statistically significant. The problem isn't running these analyses; it's cherry-picking the one that looks good after the fact and reporting it as if it were the pre-planned hypothesis.
This matters because it inflates false positive rates far beyond the nominal 5%. If you slice results by browser, device, country, and traffic source, and check significance after every slice, you'll almost certainly find something significant by chance alone — even in an A/A test with no real difference between variants.
A classic example: a test on checkout button color shows no overall lift, so the analyst breaks it down by returning vs. new visitors, then by mobile vs. desktop, then by time of day, until "mobile visitors on Tuesdays" shows a 15% lift. Reported alone, that finding looks compelling but is very likely noise.
The fix is to commit to a primary metric and analysis plan before launching, correct for multiple comparisons when you do look at several metrics, and treat any post-hoc discovery as a new hypothesis to be validated in its own follow-up test, not as a proven result.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply p-hacking.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.