Post-Hoc Analysis

Post-hoc analysis means digging into experiment data after the primary result comes in — for example, slicing an inconclusive test by browser, device, country, or new-versus-returning visitor to look for a subgroup where the variant 'won.' This is useful for generating new hypotheses, but statistically dangerous if treated as confirmatory: testing enough subgroups will eventually turn up a 'significant' result by chance alone, a pattern closely related to the multiple-comparisons problem behind false discovery rate.

The discipline here is to clearly separate pre-registered analyses (defined in the hypothesis statement before launch) from exploratory post-hoc findings, and to treat the latter as ideas to test again in a dedicated follow-up experiment rather than as proof. Reporting a post-hoc segment win as if it were the primary result is one of the more common ways experimentation programs mislead themselves and stakeholders.

Example: an overall test shows no significant lift, but a post-hoc slice reveals a large lift among users on Safari. Rather than declaring victory, the team runs a new, pre-registered experiment targeted at Safari users to confirm whether the effect replicates.