All terms

CRO Glossary

Chi-Squared Test

A statistical test that compares observed conversion counts across variants to expected counts to determine if differences are due to chance.

A chi-squared test is one of the most common statistical tools used to evaluate A/B tests with binary outcomes, like converted vs. did not convert. It compares the actual distribution of conversions across your control and variant groups against the distribution you'd expect if there were no real difference between them. If the observed numbers deviate enough from that expectation, the test flags the result as statistically significant.

Practitioners care about this because it's the engine behind most 'is this result real?' calculations in A/B testing dashboards. It's fast, doesn't require complex assumptions about underlying distributions, and works well when sample sizes are reasonably large. However, it can behave unreliably with very small sample sizes or low conversion counts, which is why some tools switch to Fisher's exact test in those cases.

For example, imagine 1,000 visitors see version A and 40 convert, while 1,000 see version B and 55 convert. A chi-squared test evaluates whether that 15-conversion gap is likely a genuine effect of the design change or just random noise, producing a p-value practitioners use to decide whether to trust the result.

It's worth remembering that chi-squared tests only tell you whether a difference exists, not how large or practically meaningful it is — that requires pairing it with effect size and confidence interval reporting.

Related terms

See this in the wild

ABWatcher watches how top teams apply chi-squared test.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.