CRO Glossary
Fisher's Exact Test
A statistical test for comparing conversion rates between two small-sample groups without relying on large-sample approximations.
Fisher's Exact Test calculates the exact probability of observing a given (or more extreme) distribution of conversions across two groups, rather than approximating it with a normal or chi-squared distribution. It's most useful when sample sizes or conversion counts are small, which is common early in a test or in low-traffic segments where the usual z-test assumptions break down.
Most A/B testing platforms default to a z-test or chi-squared test for comparing conversion rates, which works fine at scale but can produce unreliable p-values when expected cell counts (like conversions in a variant) fall below roughly 5-10. In that situation, Fisher's Exact Test gives a more trustworthy read on whether an observed difference is likely due to chance.
For example, if a checkout redesign test only ran for two days before a stakeholder wanted results, and one variant had just 6 conversions out of 40 visitors, a standard z-test could overstate significance. Running Fisher's Exact Test on that same 2x2 table (converted/not converted by variant) would give a more conservative and accurate p-value.
Practitioners care about this because misapplying large-sample tests to small samples is a quiet source of false positives in early-stage or low-traffic experiments — exactly the kind of mistake that erodes trust in a testing program over time.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply fisher's exact test.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.