CRO Glossary
Post-Hoc Analysis
Analyzing test results by slicing data into subgroups after the fact, without a pre-registered hypothesis.
Post-hoc analysis means digging into experiment data after the primary result comes in — for example, slicing an inconclusive test by browser, device, country, or new-versus-returning visitor to look for a subgroup where the variant 'won.' This is useful for generating new hypotheses, but statistically dangerous if treated as confirmatory: testing enough subgroups will eventually turn up a 'significant' result by chance alone, a pattern closely related to the multiple-comparisons problem behind false discovery rate.
The discipline here is to clearly separate pre-registered analyses (defined in the hypothesis statement before launch) from exploratory post-hoc findings, and to treat the latter as ideas to test again in a dedicated follow-up experiment rather than as proof. Reporting a post-hoc segment win as if it were the primary result is one of the more common ways experimentation programs mislead themselves and stakeholders.
Example: an overall test shows no significant lift, but a post-hoc slice reveals a large lift among users on Safari. Rather than declaring victory, the team runs a new, pre-registered experiment targeted at Safari users to confirm whether the effect replicates.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply post-hoc analysis.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.