The ABWatcher blog

Why Conversion Rates Look Better Than They Actually Are: Simpson's Paradox in CRO

A winning A/B test can be losing in every segment at once. Here's how Simpson's Paradox hides broken experiments inside a single rising conversion rate.

Sam Lee

Data Analyst · · Updated

Your dashboard says the variant is up 4%. Ship it, right? Not so fast — if you haven't cut that 4% by device, traffic source, and new-vs-returning visitor, you might be looking at a number that is mathematically true and directionally wrong. This is Simpson's Paradox, and it is one of the most common ways CRO teams declare a winner that loses in every subgroup that actually matters.

TL;DR

  • Simpson's Paradox occurs when an aggregate trend reverses once data is split by segment — confirmed mechanism per Dalton's analysis of A/B testing pitfalls
  • ABWatcher detected rotating/split hero variants at Progressive, Ally, and AB Tasty between Jun 25, 2026 and Sep 17, 2026 — all cases where aggregate read-outs would obscure segment-level behavior
  • Checkout-stage tests convert at a 32% median vs. 25% for cart-stage tests, per 2026 CRO benchmarks — pooling the two stages into one "conversion rate" already mixes populations
  • A test needs roughly 2x the sample size to detect effects reliably once split into two traffic-source segments instead of analyzed in aggregate
  • Browser-level hero rotation, like Progressive's three-way headline split, can fragment an already-thin sample into cells too small to read individually
  • Novelty effects and selection bias compound the paradox: new visitors behave differently from returning ones, and both get folded into the same top-line number by default

What exactly is Simpson's Paradox?

Simpson's Paradox is when a trend that holds in every subgroup of your data reverses or disappears once you pool the subgroups together. The classic illustration, cited in Conversion.com's rundown of invalid test mistakes, involves two kidney stone treatments: Treatment A beats Treatment B on small stones, and Treatment A beats Treatment B on large stones — yet pooled together, Treatment B appears to win. The reversal happens because the groups aren't evenly sized or evenly mixed across the subgroups.

In CRO terms: swap "treatment" for "variant" and "stone size" for "traffic source," "device," or "new vs. returning visitor," and you have the exact mechanism that turns a false winner into a shipped feature. The math isn't a bug — it's a consequence of unequal group sizes interacting with unequal baseline conversion rates. A variant can win overall purely because it happened to get more of the traffic segment that already converts better, independent of whether the variant itself did anything.

Why does your 4% conversion rate hide two different stories?

A blended conversion rate averages populations with structurally different intent, so a single number can mask one segment winning and another losing. Consider a hero test running simultaneously to paid and organic traffic. Per Digital Applied's A/B testing guide, paid traffic arrives with higher intent but lower trust, while organic visitors have usually self-researched and convert better against deeper content. If your variant happens to lean into trust signals, it may lift paid conversion while doing nothing — or actively hurting — organic. Blend the two together and you get a positive aggregate number that's really a mix of one real win and one flat-to-negative segment.

The same logic applies to funnel stage. 2026 CRO benchmarks put checkout-stage conversion at a 32% median versus 25% at the cart stage — cart abandoners have already cleared product selection and price objections, so they're a fundamentally different population than first-touch visitors. If a test's traffic mix shifts even slightly toward higher-checkout-stage visitors during the variant window, your "lift" might be a sampling artifact, not a causal effect.

How does rotating hero content create this exact trap?

Rotating or split hero content fragments traffic into cells by browser, session, or server-side logic — and if you only read the pooled number, you can't tell which cell is actually driving the result. ABWatcher caught a clean example of this at Progressive on Jun 25, 2026: Safari visitors saw "Stars, stripes, and savings" with a family outdoor scene, Chrome visitors saw "Find s'more ways to save this summer" with children, and both Firefox and Edge served "Better insurance starts here" with a family portrait. That's three distinct creative treatments split across browser — a de facto four-cell test (three variants plus whatever bucket Edge/Firefox share) hiding inside what a browser-blind analytics view would report as one aggregate conversion rate for the hero page.

The same day, ABWatcher detected a two-variant hero split at Ally: visits 1, 3, and 4 showed "Modern banking, better benefits." with a Spending Account theme and a "You just got paid 2 days sooner" phone mockup, while visit 5 showed "Open both, save more." with different subhead copy, a different hero image, and a Savings Round Up widget. Visit 2 reverted to the first variant. If Ally's internal dashboard reports one blended conversion rate across this split without conditioning on which variant the visitor actually received, any real difference between "Modern banking" and "Open both, save more" gets diluted — or worse, mis-attributed if session volume skews toward one arm for reasons unrelated to the creative itself (time of day, referral source, device mix).

Does traffic-splitting logic itself bias the segments?

Yes — when the split mechanism correlates with anything that also correlates with intent, your "random" assignment isn't actually balanced. ABWatcher's Sep 17, 2026 detection at AB Tasty is the sharpest version of this: server-side traffic splitting sent visitors to two entirely different domains — wingify.com, with soft pastel branding, "Every visitor is different" messaging, and a "Create campaign with Wingz" CTA, versus abtasty.com itself, with neon lime/purple branding and a "Push the limits of experience optimization with AI-powered conversions" hero and "Get ready for better" CTA.

This is a brand-level or product-line test, not a simple copy swap, and that raises the stakes on segment analysis considerably. If the server-side router assigns based on referral source, geography, or prior cookie state — any of which plausibly correlates with where a visitor is in their buying journey — the two arms are not comparable populations to begin with. Aggregate conversion rate on a test like this tells you almost nothing about creative performance; it mostly tells you about the routing logic.

How much sample size do you actually need to trust a segment-level read?

Splitting an aggregate test into two segments roughly doubles the sample size required to detect the same effect size at the same statistical power, because each segment now needs its own adequately powered cell. If your full test needed n=20,000 per arm to detect a 4% relative lift at 80% power and a 5% significance threshold, a single two-way segment cut (say, mobile vs. desktop) pushes the per-cell requirement toward n=20,000 again for each segment — meaning your total traffic need roughly doubles unless you extend test duration. Most teams don't budget for this, which is exactly why segment-level reversals go unnoticed: the segment cells are underpowered, so analysts default to the only statistic with enough data behind it — the aggregate.

Here's a simplified version of how an aggregate "win" can mask a segment-level loss:

Segment Control conversion Variant conversion Segment traffic share Segment verdict
Mobile 3.8% 3.5% 65% Variant loses
Desktop 5.5% 6.9% 35% Variant wins
Blended 4.3% 4.6% 100% Variant "wins" (+7%)

The blended row is the one that shows up in most dashboards by default. The mobile row — 65% of traffic, and the segment where the variant actually underperforms — is the one a shipping decision should weight most heavily, especially if mobile is a growing share of the base.

Common questions

What is Simpson's Paradox in simple terms for A/B testing?

It's when a variant appears to win on the blended conversion rate across all traffic, but actually loses (or ties) inside every individual segment — mobile, desktop, paid, organic, new vs. returning. The reversal happens because segment sizes and baseline rates differ, not because the aggregate math is wrong; it's a real statistical property, confirmed in Dalton's writeup.

How do I check if my test result is a Simpson's Paradox artifact?

Re-run your significance test on at least three cuts: device type, traffic source, and new vs. returning visitor. If the aggregate lift direction disagrees with two or more segment directions — and those segments each carry at least 20% of traffic — treat the aggregate result as unreliable until you re-power the segment cells individually.

Does novelty effect make this worse?

Yes. New visitors seeing a variant for the first time behave differently from returning visitors who've built trust with the existing design, and test duration often confounds this — early days skew toward engaged repeat visitors, later days bring in more new/cold traffic. Blending these windows into one conversion rate adds a second hidden variable on top of the segment-size problem.

Is a rotating hero (like Progressive's three-browser split) the same thing as an A/B test?

Not necessarily by design, but it has the same analytical consequence. Whether it's an intentional multivariate test or rotating creative logic, if your reporting doesn't condition on which variant a visitor received, you get the exact blended-metric distortion Simpson's Paradox describes — you just won't call it a "test" internally.

What sample size should I target to avoid underpowered segment cuts?

Start from your full-test required sample size for your minimum detectable effect, then multiply by the number of segment cells you intend to analyze independently. For a typical two-segment cut (e.g., mobile/desktop) at 80% power, budget roughly double the traffic or double the test duration versus an aggregate-only read.

Your next sprint

Before you call any test a winner, pull the conversion rate by device and by traffic source first — not after. If the direction disagrees in a segment carrying more than 20% of volume, hold the ship decision and re-power that cell specifically rather than trusting the blended number. A 4% aggregate lift on a $50M ARR page is roughly $2M annualized — which is exactly why it's worth the extra day of segment math before you bank it.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.