How to Choose the Right Statistical Test for Your A/B Test
A practical guide to picking the right statistical test—t-test, chi-square, or Fisher's exact—based on your sample size and metric type, with real live-test examples.
Aiko Tanaka
Design Director · · Updated
TL;DR
- Chi-square tests need expected cell counts of at least 5 per group; below that, use Fisher's exact test instead.
- Two-proportion z-tests work for binary conversion metrics (converted/didn't) with sample sizes typically above 1,000 visitors per arm.
- Welch's t-test (not Student's) is the safer default for continuous metrics like AOV or session duration when variances differ between groups.
- Standard significance calculators assume equal variance and large-sample normality — assumptions that break down under 100 conversions per variant.
- ABWatcher detected a cookie-bucketed pricing-page test at Disney+ on 20 Jul 2026 that a naive calculator would have mis-segmented by browser engine alone.
- A layout test at Hotels.com, detected 15 Aug 2026, mixed a new merchandising section with copy changes — a case where a single significance test can't isolate which element moved the metric.
Why does test selection matter more than the calculator you use?
Because the wrong test gives you a p-value that looks precise but answers the wrong question. Most teams paste conversion counts into a generic significance calculator, get a green checkmark or a red X, and ship the decision without asking whether the underlying math actually fits their data. A two-proportion z-test, a chi-square test, and Fisher's exact test can all claim to test "is variant B better than variant A" — but they make different assumptions about sample size, cell counts, and distribution shape. Pick the wrong one and you can manufacture false confidence in a result that's actually noise, or bury a real lift under an overly conservative test.
This matters even more now that experimentation has gone mainstream: research shows A/B testing can lift conversion rates by 20% to 49% when teams test the right elements — but that number assumes the underlying stats are sound. A beautifully designed test with a broken significance test is still a broken test.
Which statistical test fits your metric type?
The test you choose depends on whether your metric is binary, continuous, or count-based, and how many observations you have per variant.
| Metric type | Example | Typical test | Minimum sample guidance |
|---|---|---|---|
| Binary (converted / not) | Sign-up rate, checkout completion | Two-proportion z-test or chi-square | 5+ expected conversions per cell; 1,000+ visitors/arm recommended |
| Binary, small sample | New feature click-through in a soft launch | Fisher's exact test | Works even with single-digit cell counts |
| Continuous | AOV, time-on-page, session length | Welch's t-test | 30+ observations per arm for CLT to hold reasonably |
| Count/rate | Support tickets per user, emails sent | Poisson or negative binomial regression | Depends on event rate; often needs weeks of data |
| Multiple variants | 3+ hero layouts tested simultaneously | ANOVA or chi-square with correction | Account for multiple comparisons (Bonferroni, Holm) |
A chi-square test assumes every expected cell count is 5 or higher. If your lowest-traffic variant only logged 3 conversions out of 40 visits, chi-square's approximation breaks down and the p-value becomes unreliable. That's exactly when you switch to Fisher's exact test, which calculates the exact probability of your observed table rather than relying on a large-sample approximation. Fisher's exact is more conservative but it's correct at small n, where chi-square simply isn't.
When do standard significance calculators fall short?
Standard calculators fall short when your traffic is segmented, your variance is unequal, or your test ran across more than two arms. Most free calculators assume a clean 50/50 split, independent observations, and roughly equal variance between groups — assumptions that real traffic rarely respects.
Take the pricing test ABWatcher detected at Disney+ on 20 Jul 2026. The "Choose your plan" section rendered as a card layout with SELECT buttons in some visits, and a text/checkmark comparison table in others. Critically, the split didn't follow browser engine lines — Edge (Blink) showed the table variant like Firefox, not matching Chrome. That's the signature of cookie-based bucketing, not a rendering artifact. If you tried to analyze this test by segmenting your calculator inputs by browser, you'd contaminate both arms with the wrong users and your significance test would be measuring noise, not the actual experiment. The lesson: know your bucketing mechanism before you pick a test, not after.
Student's t-test is another common default that quietly fails. It assumes equal variance between control and variant. In practice, a redesigned page often doesn't just shift the mean — it changes the spread. Welch's t-test relaxes the equal-variance assumption and should be your default for any continuous metric comparison unless you've explicitly verified variances are close (an F-test or Levene's test can confirm this in under a minute).
How do you handle tests that change more than one thing at once?
You isolate the metric each element is most likely to move, and avoid attributing a single lift to a bundle of changes. This is a design-craft problem as much as a stats problem.
Look at the Hotels.com layout test detected 15 Aug 2026. The variant added an entire "Last-minute deals, right on time" section with pricing and reviews — new visual weight competing for attention above the fold — while simultaneously swapping the sign-in prompt from "Unlock instant savings with Member Prices" to a plainer "Sign in with your email" copy. If both changes shipped in a single variant, a single significance test on overall conversion rate can't tell you whether the lift (or drop) came from the new merchandising module, the softer sign-in copy, or the interaction between the two competing calls to attention. The honest move is either to test these independently, or to define secondary metrics — click-through on the new deals section, sign-in completion rate — and run separate tests for each, each with its own appropriately chosen statistical test, rather than one global number covering two different design decisions.
The Attio test detected 11 Jul 2026 has the same shape: the two variants reorder the same content blocks (AI feature cards, logos, pricing, testimonials, FAQ) and relabel sections — "Loved by thousands of companies" with card quotes versus a tweet-style testimonial grid. Reordering changes what hits the eye first under F-pattern reading; a tweet-style grid reads as higher-velocity social proof than static cards, which could shift scroll depth before it shifts conversion. If you're only tracking bottom-funnel conversion, you may need thousands more sessions to detect that effect than you would for a metric closer to the change, like scroll-to-pricing rate.
How does copy-only testing change your sample size math?
Copy tests usually need larger samples than layout tests because the expected effect size is smaller. A subtle wording change rarely moves conversion by more than a percentage point or two, so your test needs more statistical power to detect it reliably.
The Lyft test detected 8 Jul 2026 is a clean example: the "Choose how to ride" subheadline varied between "Find a ride that suits your mood and budget" and "Hit the road, or the rails. Find a ride that fits your style and budget" — and this split showed up within the same browser engine (Chrome), ruling out a rendering artifact and confirming a genuine bucketed experiment. Two phrasings of a supporting subheadline, several rungs below the primary CTA in visual hierarchy, are unlikely to move conversion by more than a point or two. A minimum detectable effect (MDE) calculation should come before the test launches: plug in your baseline conversion rate, desired power (0.8 is standard), and significance threshold (usually 0.05), and solve for the sample size needed to detect a 1-2% relative lift. For most mid-traffic sites, that means running the test for multiple weeks, not days — experimentation platforms increasingly let teams set confidence thresholds per test rather than forcing every test through one default framework, which is exactly the flexibility a subheadline test needs versus a full-funnel redesign.
Contrast that with a bigger structural swing, like the SignNow hero test detected 8 Aug 2026, where one variant collapsed the layout to a single "Get your first document signed" CTA with no logo strip, and the other added a two-column choice architecture plus a client-logo bar (Walmart, ExxonMobil, Apple, Comcast, Facebook, FedEx) and a comparison callout. That's a much larger expected effect — new social proof, new decision architecture, more visual weight added to the fold — so it needs a smaller sample to reach significance than a one-word subheadline swap would. Matching test magnitude to sample size expectations up front saves you from either stopping too early on a subtle change or over-running a test that was always going to show an obvious result.
Common questions
When should I use Fisher's exact test instead of chi-square?
Use Fisher's exact test when any expected cell count in your contingency table is below 5 — common in early-stage tests, low-traffic pages, or tests segmented into many variants. Chi-square's approximation becomes unreliable at small counts; Fisher's exact calculates exact probabilities and stays valid regardless of sample size.
Can I just use a 95% confidence calculator for every test?
No. A 95% threshold calculator typically assumes a two-proportion z-test with roughly equal variance and sample sizes in the hundreds or thousands. For continuous metrics, small samples, or multi-arm tests, you need a different test entirely — otherwise the "95% confidence" figure doesn't mean what the tool implies.
How do I know if my A/B test was bucketed by cookie versus browser?
Check whether variant assignment correlates with browser engine (Chrome/Blink vs. Firefox vs. Safari) across repeat visits. If the same browser engine shows different variants on different visits — as seen in the Disney+ pricing test detected 20 Jul 2026 — that points to cookie- or session-based bucketing, not a rendering difference.
What sample size do I need before trusting a t-test result?
Aim for at least 30 observations per arm as a rule of thumb for the Central Limit Theorem to approximate normality reasonably well, though skewed metrics like revenue often need hundreds more. Below 30, consider a non-parametric alternative like the Mann-Whitney U test instead of a standard t-test.
Does testing multiple elements at once require a different statistical approach?
Yes. When a variant bundles multiple changes — like the Hotels.com test that changed both a merchandising section and sign-in copy — you should define separate metrics per element and consider factorial or multivariate test designs rather than relying on one aggregate significance test to explain everything that moved.
The takeaway for your next sprint
Before your next test ships, write down three things: the metric type (binary, continuous, or count), your expected sample size per arm, and the minimum detectable effect you actually care about. That combination tells you which test to run — not the other way around. If you're testing a subtle copy change like Lyft's subheadline, budget for a longer run and a larger sample. If you're testing a structural change like SignNow's hero, expect to reach significance faster, but double-check your bucketing logic the way the Disney+ and Hotels.com detections remind us to. Put the right test next to the right data, and your significance results will actually mean what you think they mean.
Read next
More from the ABWatcher blog
Nov 22, 2026
How to Calculate Confidence Intervals for Conversion Rate Lift
A practical guide to computing confidence intervals for A/B test conversion lift — frequentist and Bayesian methods, worked examples, and how to report ranges instead of fake certainty.
Nov 21, 2026
Why Your Conversion Rate Varies by Traffic Source (And What to Do About It)
A diagnostic guide to conversion rate variance across organic, paid, and referral traffic — how to separate real signal from source-quality noise before you spin up a new test.
Nov 20, 2026
Why Your A/B Test Winners Don't Replicate Across Traffic Sources
A test winner that holds for blended traffic often fails for a specific channel. Here's why, with real examples, and how to validate before you scale.
Nov 19, 2026
Session Recording for CRO: What to Watch When Users Bounce
A practical, intro-level guide to reading session replays for hidden friction and high-intent signals — plus how to turn what you see into testable hypotheses.
Nov 18, 2026
How to Build a Winning Landing Page Without A/B Testing
No traffic for statistical significance? Here's a design-led framework for building high-converting landing pages using proven patterns, not guesswork.
Nov 17, 2026
How to Find Your True Conversion Bottleneck (Not Just the Waterfall)
A step-by-step method for pinpointing the real cause of funnel drop-off using session recordings, heatmaps, and field-level friction data — before you waste a test cycle.
From the glossary
Vocabulary the article uses. See all terms →
- Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
- Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
- Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
- Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
- Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
- Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See more like this
ABWatcher catches A/B tests like this across 1,000+ brands.
Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.