P-hacking

P-hacking happens when someone runs a test, doesn't get the result they hoped for, and then keeps tweaking things — adding segments, removing outliers, extending the test, or checking dozens of secondary metrics — until something turns up statistically significant. The problem isn't running these analyses; it's cherry-picking the one that looks good after the fact and reporting it as if it were the pre-planned hypothesis.

This matters because it inflates false positive rates far beyond the nominal 5%. If you slice results by browser, device, country, and traffic source, and check significance after every slice, you'll almost certainly find something significant by chance alone — even in an A/A test with no real difference between variants.

A classic example: a test on checkout button color shows no overall lift, so the analyst breaks it down by returning vs. new visitors, then by mobile vs. desktop, then by time of day, until "mobile visitors on Tuesdays" shows a 15% lift. Reported alone, that finding looks compelling but is very likely noise.

The fix is to commit to a primary metric and analysis plan before launching, correct for multiple comparisons when you do look at several metrics, and treat any post-hoc discovery as a new hypothesis to be validated in its own follow-up test, not as a proven result.