The ABWatcher blog

Segmentation in A/B Tests: How to Avoid Simpson's Paradox

Aggregate A/B test results can hide opposite effects by segment. Here's how Simpson's Paradox shows up in live tests and how to segment correctly.

Jordan Reeves

Founder & Operator · · Updated

I shipped a "winning" pricing test once that lifted revenue per visitor by 6% overall. Three weeks later our finance lead pulled up the cohort breakdown and the test was actually losing money on every segment that mattered long-term — it had only won because it attracted a flood of low-LTV trial signups that inflated the top-line number before anyone looked deeper. Nothing was wrong with our stats engine. The aggregate was true. It was also completely misleading. That's Simpson's Paradox, and if you've run more than a dozen tests, you've probably already shipped a loser dressed up as a winner.

TL;DR

  • Simpson's Paradox occurs when a test's aggregate result reverses or hides opposite effects present in underlying segments — common in tests with uneven traffic mix.
  • Dalton's research recommends checking device type, traffic source, and new-vs-returning status for every test before acting on results.
  • Properly segmented A/B tests can increase conversion lift by 20% to 49% when teams test page elements and messaging by audience.
  • ABWatcher detected a live pricing-plan test at Attio on 8 Sep 2026 showing two different price points ($29 vs $35) served to different sessions.
  • ABWatcher detected a hero-and-banner test at Ally on 17 Sep 2026 where two distinct headline/promo combinations appeared across the same browser engine.
  • If segment winners disagree with the aggregate winner, that's the signal to investigate before shipping — not a reason to ignore the data.

What is Simpson's Paradox and why does it break A/B tests?

Simpson's Paradox is when a trend that holds across combined data disappears or reverses when the data is split into natural subgroups. In A/B testing terms: Variant B can "win" overall while actually losing with mobile users, losing with paid traffic, and losing with returning customers — and only winning because of how traffic happened to be distributed across those groups, not because of anything the variant did well.

This isn't rare. It's the mathematical consequence of three conditions that show up in almost every real test: segments convert at different baseline rates, segments receive different traffic volumes, and the variant effect isn't uniform across segments. Put those three together and your top-line number can tell you the opposite of what's actually happening underneath.

How does segment traffic mix create false winners?

A false winner emerges when one segment both converts higher and makes up a larger share of one variant's traffic, masking a loss in every other segment. Picture a checkout test where 70% of Variant A's traffic happened to be returning customers (who convert at 18% regardless of variant) while Variant B skewed toward new visitors (who convert at 4%). Variant A looks like it's crushing it. In reality, among new visitors, B was winning by two points, and among returning visitors, there was no difference at all. The aggregate lift is a traffic-mix artifact, not a product improvement.

This is exactly the kind of thing a pricing-page test can hide. ABWatcher caught Deel running a pricing-page A/B test detected 17 Sep 2026, where the variant swapped the hero CTA from "Book a demo" to "Speak with a trusted expert," dropped the Bundle tab entirely, and rewrote plan-card copy into longer, benefit-style descriptions. A test with this many simultaneous changes is a segmentation minefield: enterprise buyers evaluating the Bundle tab specifically will respond completely differently than SMB self-serve visitors who never looked at it. Aggregate conversion on a change like this tells you almost nothing about which buyer type actually moved.

Which segments should you actually check before shipping a test?

At minimum, check device type, traffic source, and new-versus-returning status before declaring a winner. This is the baseline cut recommended by Dalton's analysis of Simpson's Paradox in CRO: if segment-level winners agree with the aggregate winner, ship with confidence. If they diverge, stop and dig in before you roll out to 100%.

Here's the segmentation checklist I actually run on every test above a certain traffic threshold:

Segment dimension Why it matters Minimum sample to trust it
Device type (mobile/desktop/tablet) Layout and friction differ wildly by surface ~1,000 conversions per cell
Traffic source (paid/organic/direct/referral) Intent and price sensitivity vary by channel ~800 conversions per cell
New vs. returning visitor Returning users already trust the brand; new users are evaluating it ~1,000 conversions per cell
Geography / currency Pricing and plan tests often perform opposite by region ~500 conversions per cell
Plan tier or cohort (for SaaS) Enterprise and self-serve buyers react to different levers ~300 conversions per cell

You don't need all five on every test. But if you're testing anything touching price, plan structure, or a high-stakes CTA, skip the first two at your own risk.

What do real segmentation traps look like in the wild?

Live tests we've tracked show the masking effect happening right now across pricing, hero, and form surfaces. ABWatcher detected an Attio pricing test on 8 Sep 2026 where Safari sessions saw two structurally different plan tables — Plus priced at $29 versus $35, Pro at $69 versus $79, with an entire "Seats" row and "MCP server" row present in only one version. A test with this much structural variance between variants almost guarantees different reactions by company size: a five-person startup evaluating seat limits will respond nothing like a 200-person team evaluating the MCP server row. If Attio (or any team running a test like this) only looks at blended conversion rate, they'll miss exactly which segment is driving the number either direction.

The Ally hero test detected 17 Sep 2026 is a cleaner illustration of why source and timing matter as a segment too, not just device. Two distinct headline-and-banner combinations rotated across sessions regardless of browser engine — "More bank for your buck" paired with a WNBA promo banner in one variant, "The bank fans bank on" paired with different messaging in the other. Visitors arriving from a sports-content referral versus a finance-comparison site are going to read "Bank on Buckets" completely differently. Treating all traffic as one pool erases that signal entirely.

Even something as low-stakes as an email capture popup isn't immune. ABWatcher caught Beis running a form test detected 8 Sep 2026 where the same 20%-off offer was shown with two different lifestyle images — luggage on the floor versus luggage in a vintage convertible — alongside a footer CTA change from "SIGN UP FOR TEXTS" to "SIGN UP NOW." Image preference is notoriously audience-dependent: a convertible-and-travel aesthetic might convert significantly better with a leisure-travel segment and worse with a business-travel segment buying the same product. Aggregate lift on a test like this can easily be zero while masking a real preference split underneath.

And redirect-style tests carry their own version of this trap. The AB Tasty hero test detected 17 Sep 2026 sent visitors to two entirely different destination sites — wingify.com with soft pastel branding and "Every visitor is different" messaging, versus abtasty.com with neon branding and an AI-optimization pitch. When a test routes to fully separate brand experiences, you're not just testing a headline, you're testing brand positioning against two different buyer psychologies. That only becomes visible once you segment by how visitors arrived and what they were looking for.

How do you segment without destroying your sample size?

The fix is pre-registering two or three segment cuts before the test launches, not fishing through dozens of slices afterward. Decide upfront — device, source, new/returning, maybe one more — and commit to checking only those. Post-hoc segment hunting is how you manufacture false positives in the other direction, where you slice data until some subgroup shows a "significant" result by chance.

Practical minimums: don't trust a segment cut with fewer than a few hundred conversions per cell, and don't run more than three or four pre-registered cuts per test. If you need finer granularity, that's a sign you need more traffic or a longer test window, not more slicing. Field-tested CRO guidance backs this up — the teams that get reliable reads are the ones who commit to a measurement plan before launch, not the ones with the fanciest dashboard after the fact.

For SaaS teams, the highest-value cut is almost always plan tier or company size, because self-serve and enterprise buyers respond to completely different levers — Attio's seat-limit and MCP-row variance is a textbook case. For ecommerce, the highest-value cut is usually new-vs-returning plus device, because first-time buyers on mobile are a different animal entirely from loyal desktop shoppers, as the Beis image-variant test implicitly tests whether you check it or not.

Common questions

What's the simplest way to spot Simpson's Paradox in my own test results?

Split your results by device and by new-vs-returning visitor and compare winners in each slice to the aggregate winner. If any segment's winner disagrees with the overall winner, you likely have a masking effect. This takes ten minutes in most analytics tools and should be standard practice before any rollout decision.

How much traffic do I need before segmenting is trustworthy?

Aim for at least a few hundred conversions per segment cell, not just per variant. A test with 2,000 total conversions split 50/50 across four segments only gives you roughly 250 conversions per cell, which is thin — treat those reads as directional, not conclusive, and extend the test if the decision is high-stakes.

Should I ever ship a test that wins in aggregate but loses in a key segment?

Only if that segment is explicitly low-priority for your business and you've confirmed the loss won't compound (e.g., through churn or support cost) later. Otherwise, treat a segment-level loss in a high-value cohort as a blocking issue, even if the topline number looks like a clean win.

Does segmenting by traffic source actually change test outcomes often?

Yes — paid and organic visitors frequently show opposite reactions to pricing and CTA changes because they arrive with different intent and price anchoring. Tests like Deel's pricing-page variant, which alters both CTA language and plan structure simultaneously, are especially prone to source-driven reversals.

Is it ever okay to skip segmentation entirely?

Only for very low-stakes, easily reversible tests — a button color or minor copy tweak with no pricing or structural implications. Anything touching price, plan tiers, or core messaging (the kind of tests we track at Attio, Deel, and Ally) should always get at least the three baseline cuts before you call it.

What I would actually do next

Before your next test review meeting, pull up your last three "winning" tests and re-run the numbers split by device and by new-vs-returning visitor only — don't boil the ocean with ten segments. If any of those three tests shows a different winner in a key segment than it did in aggregate, you've just found a false conclusion that's probably still live on your site right now. Fix that one test this sprint before you greenlight anything new.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.