CRO Glossary
Above Threshold Significance Chasing
The practice of tweaking a live test's audience, duration, or metric definition until the result crosses a significance threshold.
Significance chasing is when a team keeps adjusting an in-flight or just-finished experiment — narrowing the segment, extending the runtime, swapping the success metric, or excluding 'noisy' traffic — specifically because the original result fell short of significance, and they want to find a cut of the data that clears the bar. It's related to p-hacking but focuses specifically on the post-hoc fiddling with test parameters rather than just statistical cherry-picking of metrics.
This matters because it silently erodes the validity of experimentation as a decision-making tool. Once a team learns that an inconclusive test can be rescued by filtering to mobile users only, or by re-running the analysis after excluding the first three days, the entire program's results become suspect. The test stops answering 'does this change work' and starts answering 'can we find a story that lets us ship what we already wanted to ship.'
A concrete example: a redesigned pricing page shows a flat result at two weeks. Rather than accepting the null result, the team re-segments by new-vs-returning visitors, finds a positive effect in new visitors only, and declares victory — ignoring that they tested a dozen segments and found one significant by chance. Had they pre-registered that segment as the primary metric, it would be valid; discovering it after the fact and presenting it as the headline result is the problem.
The defense is procedural: lock the success metric, segment, and stopping rule before launch, and treat any post-hoc exploration as hypothesis-generating for a future test, not as evidence to act on today.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply above threshold significance chasing.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.