All terms

CRO Glossary

Fisher's Exact Test

A statistical test for comparing conversion rates between two small-sample groups without relying on large-sample approximations.

Fisher's Exact Test calculates the exact probability of observing a given (or more extreme) distribution of conversions across two groups, rather than approximating it with a normal or chi-squared distribution. It's most useful when sample sizes or conversion counts are small, which is common early in a test or in low-traffic segments where the usual z-test assumptions break down.

Most A/B testing platforms default to a z-test or chi-squared test for comparing conversion rates, which works fine at scale but can produce unreliable p-values when expected cell counts (like conversions in a variant) fall below roughly 5-10. In that situation, Fisher's Exact Test gives a more trustworthy read on whether an observed difference is likely due to chance.

For example, if a checkout redesign test only ran for two days before a stakeholder wanted results, and one variant had just 6 conversions out of 40 visitors, a standard z-test could overstate significance. Running Fisher's Exact Test on that same 2x2 table (converted/not converted by variant) would give a more conservative and accurate p-value.

Practitioners care about this because misapplying large-sample tests to small samples is a quiet source of false positives in early-stage or low-traffic experiments — exactly the kind of mistake that erodes trust in a testing program over time.

Related terms

See this in the wild

ABWatcher watches how top teams apply fisher's exact test.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.