Why Your A/B Test Results Contradict Between Segments
Your test has a clear winner overall — but mobile says the opposite of desktop. Here's how Simpson's Paradox creeps into live A/B tests, and how to catch it before you ship.
Jordan Reeves
Founder & Operator · · Updated
TL;DR
- Simpson's Paradox occurs when an aggregate A/B test winner reverses or disappears once you split by segment — traffic source, device, or new vs. returning.
- Ally ran a hero/banner test detected 17 Sep 2026 with two distinct headline-banner pairings surfacing inconsistently across identical browser engines, a classic setup for segment contradictions.
- Buffer's hero subheadline test (detected 26 Aug 2026) split cleanly across both Chrome and Firefox sessions, which is good practice — it ruled out browser as a confound before segment analysis even started.
- Snake River Farms' email-capture/exit-intent form test (detected 8 Aug 2026) showed bucket-driven divergence within the same browser, the kind of unequal group composition that breeds paradox results.
- Fix the paradox by pre-registering your segment cuts before launch, not after you see a result you don't like.
What is Simpson's Paradox in A/B testing?
Simpson's Paradox is when a test's overall result reverses once you break the data into subgroups. You ship Variant B because it beat Variant A by 8% in aggregate. Three weeks later someone on the data team slices it by traffic source and finds Variant A actually won among paid traffic, organic traffic, and returning visitors — individually, every single segment. B only "won" in aggregate because the segments mixing ratio shifted between the two groups. I've shipped this mistake. Not once. The fix isn't more statistics — it's a design habit.
Here's the mechanical version: if your control group happened to get 70% mobile traffic and your variant group got 55% mobile traffic — because of imperfect randomization, a mid-test campaign launch, or a bot-filtering rule that fired differently — your "winner" might just be an artifact of which segment got more weight in each bucket. The individual segments can each prefer the loser and the paradox still shows the loser as an aggregate winner. It's not a rare statistical curiosity. It's what happens anytime segment mix drifts between groups and segments have meaningfully different baseline conversion rates.
Why does it actually happen on live sites?
It happens because traffic composition is never static during a live test window, and most teams only check the top-line number. A test that runs across a product launch, a paid campaign spike, a seasonal spend shift, or even a time-zone-driven traffic wave will naturally change its segment mix mid-flight — even with proper randomization at the user level, the composition of who shows up during each phase of the test can differ if your sample ratios shift or your targeting rules interact with the variant assignment logic.
Consider a test like Ally's hero/banner experiment, detected 17 Sep 2026. Two distinct pairings surfaced: "More bank for your buck" with a WNBA "Bank on Buckets" promo banner in one version, versus "The bank fans bank on" with a "Chase your money dreams" banner in the other. What's notable from a Simpson's Paradox lens isn't the creative — it's that both variants appeared within the same browser engine, with Chrome sessions alone showing both combinations at different visit ratios (2/5 vs 6). When a sports-sponsorship-themed banner is live, you're almost certainly drawing a traffic composition shift — people clicking through from WNBA-adjacent placements behave differently than people landing from a generic search query. If that referral spike isn't evenly distributed across both variant buckets, your aggregate winner could easily be a composition artifact, not a creative one.
Contrast that with Buffer's subheadline test, detected 26 Aug 2026. The split — "Connected to every platform and tool you use" versus "Works with every platform you post to, and plugs into your favorite tools" — appeared across both Chrome and Firefox sessions on both sides of the test. That's the behavior you want to see: it confirms the split is driven by bucketing logic, not by some confound like browser rendering quirks or ad-blocker interference skewing one engine toward one variant. It doesn't prove the result is paradox-free, but it removes one obvious lurking variable before you even get to segment analysis.
How do you detect it before you ship?
You detect it by running your segment breakdown before you declare a winner, not after. Here's the actual checklist I use:
- Pre-register 3-4 segment cuts before the test launches — typically new vs. returning, mobile vs. desktop, and top two traffic sources by volume. Write them down. Don't cherry-pick cuts after seeing results; that's how you p-hack your way into a false paradox "discovery" or miss a real one.
- Check segment mix ratio daily, not just at the end. If control is 62% mobile on day 1 and 71% mobile on day 10, something is shifting your sample — a campaign, a bug in the bucketing script, or a seasonal pattern.
- Weight your segments if the mix is structurally unequal. If mobile always converts at half the rate of desktop and your test naturally skews traffic differently by variant (common with mobile-specific creative), a simple weighted average by segment share will catch what raw aggregate hides.
- Look for bucket-driven anomalies within a single environment. Snake River Farms ran a form test detected 8 Aug 2026 where one Chrome visit omitted the mid-page "$25 off" email capture block and exit-intent popup entirely, while two other Chrome visits on the same engine matched the control layout. That's a signal worth chasing — when the same browser, same rough visitor type, shows divergent experiences, it's your bucketing logic creating segment inequality, and it's exactly the kind of invisible split that produces paradox results three weeks later.
This matches what the broader A/B testing literature on segmentation flags as a top-tier mistake: focusing on overall CVR ignores that new visitors, returning users, and mobile users each have fundamentally different behaviors, and treating them as one blended number hides exactly where your real signal — or your real paradox — lives.
Which test surfaces are most prone to this problem?
Hero sections and conversion-critical form flows are the highest-risk surfaces because they interact heavily with traffic source and intent. A hero test is especially exposed because it's usually the first thing every visitor sees, which means every traffic-source shift, every campaign launch, and every seasonal spike flows directly through it. Marcus's hero test, detected 30 Jun 2026, is a good example of the stakes: control showed a "See how you compare to other banks" interactive tool, while the variant swapped in a "Start Saving with our high-yield rate" layout routed through a completely separate funnel (confirmed via server-side routing to distinct high-yield-savings paths). Two structurally different funnels, each likely to appeal to different visitor intents — someone already comparison-shopping versus someone ready to act — means segment composition (shopping-stage visitors vs. ready-to-convert visitors) could easily produce opposite winners depending on which intent group dominates your sample that week.
Social proof and trust elements carry a milder but real version of the same risk. Atlas Coffee Club's footer test, detected 18 Aug 2026, swapped an email-capture CTA ("Your first bag free") against a 5-star customer testimonial in the same footer slot. Email-capture CTAs tend to perform differently for first-time visitors than for people who've already seen the brand's testimonials elsewhere in their journey — meaning a paid-traffic cold audience and an organic-retargeted audience could reasonably disagree on which footer treatment converts better, and an aggregate blend would mask that disagreement entirely.
| Surface | Example detection | Paradox risk driver |
|---|---|---|
| Hero / headline | Ally, 17 Sep 2026 | Campaign-driven traffic spikes shift segment mix mid-test |
| Hero / subheadline | Buffer, 26 Aug 2026 | Lower risk — split confirmed consistent across browser engines |
| Footer / social proof | Atlas Coffee Club, 18 Aug 2026 | Cold vs. warm audience intent differs on trust-signal response |
| Form / exit-intent | Snake River Farms, 8 Aug 2026 | Bucket-level inconsistency within same browser/device |
| Hero / funnel routing | Marcus, 30 Jun 2026 | Distinct funnels appeal to different intent segments |
How do you design tests that avoid the paradox entirely?
You avoid it by fixing your primary metric, your segment cuts, and your minimum sample-per-segment before a single visitor is bucketed. The most common root cause, according to most A/B testing guides, is bundling too many changes into one variant and picking a primary metric that doesn't reflect the real business lever — click rate instead of completed purchase, aggregate conversion instead of revenue per visitor. Add segment imbalance on top of a fuzzy metric and you get a result that looks confident and is actually noise wearing a suit.
Practical guardrails that actually hold up:
- Stratified randomization. Bucket users within each segment (mobile/desktop, new/returning) separately so each variant gets a proportionally equal slice of each segment, not just equal totals.
- Run time matters more than sample size. A test that hits "significance" in four days almost always caught a temporary traffic-composition wave, not a stable preference. Run through at least one full weekly cycle, ideally two.
- Decide your minimum segment sample size upfront. If a segment won't hit 300-500 conversions during the test window, don't report a segment-level verdict on it — just flag it as inconclusive rather than let it skew the aggregate narrative.
- Hold the metric constant across the whole analysis. If you launched the test to measure purchase completion, don't quietly switch to click-through when purchase doesn't show the result you expected. That's exactly the kind of metric-shopping that manufactures a false paradox.
Common questions
What's a simple example of Simpson's Paradox in A/B testing?
Imagine Variant A converts better among both mobile and desktop users individually, but Variant B "wins" in aggregate because the test accidentally sent more mobile traffic (a lower-converting segment) to Variant A. The aggregate number is real, but it reflects mix, not creative performance — segment-level data tells the true story.
How many segments should I check before trusting a result?
Three to four is usually enough: new vs. returning, mobile vs. desktop, and your top one or two traffic sources by volume. More than that and you risk p-hacking — finding a "significant" segment purely by checking enough subgroups until one shows noise. Pre-register the cuts before launch.
Does a longer test duration fix Simpson's Paradox?
It helps but doesn't guarantee it. A longer run smooths out short-term traffic-composition spikes (a campaign, a viral post) but won't fix structural imbalance from the bucketing logic itself. Pair a full 1-2 week run with stratified randomization for a real fix, not duration alone.
Is Simpson's Paradox more common in ecommerce or SaaS tests?
It's common wherever traffic sources vary widely in intent and conversion rate, which hits ecommerce especially hard around promotions and seasonal campaigns. SaaS tests see it too, particularly on pricing and hero pages where paid vs. organic visitors have very different buying stages, as seen in funnel-routing tests like Marcus's.
Can I fix a paradox after the test has already ended?
Partially. You can re-run segment-level analysis on existing data to catch a mix problem retroactively, but you can't un-bias a poorly stratified sample. The honest move is to flag the result as inconclusive, fix the bucketing logic, and re-run — shipping on a known-paradoxical result is worse than admitting the test needs a rerun.
What I would actually do next
Before your next test ships, pull up the last three "winning" tests your team has shipped and run the segment breakdown you skipped the first time — mobile vs. desktop, new vs. returning, top traffic source vs. everything else. If any of them flip or flatten at the segment level, don't panic and don't revert blindly; just add stratified randomization and a pre-registered segment checklist to your next test brief this sprint. That one habit — checking the subgroup story before you trust the aggregate headline — is the cheapest insurance you'll ever buy against shipping the wrong variant to 100% of your traffic.
Read next
More from the ABWatcher blog
Nov 22, 2026
How to Calculate Confidence Intervals for Conversion Rate Lift
A practical guide to computing confidence intervals for A/B test conversion lift — frequentist and Bayesian methods, worked examples, and how to report ranges instead of fake certainty.
Nov 21, 2026
Why Your Conversion Rate Varies by Traffic Source (And What to Do About It)
A diagnostic guide to conversion rate variance across organic, paid, and referral traffic — how to separate real signal from source-quality noise before you spin up a new test.
Nov 20, 2026
Why Your A/B Test Winners Don't Replicate Across Traffic Sources
A test winner that holds for blended traffic often fails for a specific channel. Here's why, with real examples, and how to validate before you scale.
Nov 19, 2026
Session Recording for CRO: What to Watch When Users Bounce
A practical, intro-level guide to reading session replays for hidden friction and high-intent signals — plus how to turn what you see into testable hypotheses.
Nov 18, 2026
How to Build a Winning Landing Page Without A/B Testing
No traffic for statistical significance? Here's a design-led framework for building high-converting landing pages using proven patterns, not guesswork.
Nov 17, 2026
How to Find Your True Conversion Bottleneck (Not Just the Waterfall)
A step-by-step method for pinpointing the real cause of funnel drop-off using session recordings, heatmaps, and field-level friction data — before you waste a test cycle.
From the glossary
Vocabulary the article uses. See all terms →
- Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
- Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
- Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
- Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
- Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
- Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See more like this
ABWatcher catches A/B tests like this across 1,000+ brands.
Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.