All terms

CRO Glossary

Above Threshold Significance Chasing

The practice of tweaking a live test's audience, duration, or metric definition until the result crosses a significance threshold.

Significance chasing is when a team keeps adjusting an in-flight or just-finished experiment — narrowing the segment, extending the runtime, swapping the success metric, or excluding 'noisy' traffic — specifically because the original result fell short of significance, and they want to find a cut of the data that clears the bar. It's related to p-hacking but focuses specifically on the post-hoc fiddling with test parameters rather than just statistical cherry-picking of metrics.

This matters because it silently erodes the validity of experimentation as a decision-making tool. Once a team learns that an inconclusive test can be rescued by filtering to mobile users only, or by re-running the analysis after excluding the first three days, the entire program's results become suspect. The test stops answering 'does this change work' and starts answering 'can we find a story that lets us ship what we already wanted to ship.'

A concrete example: a redesigned pricing page shows a flat result at two weeks. Rather than accepting the null result, the team re-segments by new-vs-returning visitors, finds a positive effect in new visitors only, and declares victory — ignoring that they tested a dozen segments and found one significant by chance. Had they pre-registered that segment as the primary metric, it would be valid; discovering it after the fact and presenting it as the headline result is the problem.

The defense is procedural: lock the success metric, segment, and stopping rule before launch, and treat any post-hoc exploration as hypothesis-generating for a future test, not as evidence to act on today.

Related terms

See this in the wild

ABWatcher watches how top teams apply above threshold significance chasing.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.