CRO Glossary
Chi-Squared Test
A statistical test that compares observed conversion counts across variants to expected counts to determine if differences are due to chance.
A chi-squared test is one of the most common statistical tools used to evaluate A/B tests with binary outcomes, like converted vs. did not convert. It compares the actual distribution of conversions across your control and variant groups against the distribution you'd expect if there were no real difference between them. If the observed numbers deviate enough from that expectation, the test flags the result as statistically significant.
Practitioners care about this because it's the engine behind most 'is this result real?' calculations in A/B testing dashboards. It's fast, doesn't require complex assumptions about underlying distributions, and works well when sample sizes are reasonably large. However, it can behave unreliably with very small sample sizes or low conversion counts, which is why some tools switch to Fisher's exact test in those cases.
For example, imagine 1,000 visitors see version A and 40 convert, while 1,000 see version B and 55 convert. A chi-squared test evaluates whether that 15-conversion gap is likely a genuine effect of the design change or just random noise, producing a p-value practitioners use to decide whether to trust the result.
It's worth remembering that chi-squared tests only tell you whether a difference exists, not how large or practically meaningful it is — that requires pairing it with effect size and confidence interval reporting.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply chi-squared test.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.