The Null Hypothesis in A/B Testing: A No-Jargon Guide
A plain-English guide to the null hypothesis in A/B testing, with real live-test examples from Magic Spoon, AB Tasty, Beis, Attio, and Ally.
Aiko Tanaka
Design Director · · Updated
Most teams don't lose money because their A/B test tool broke. They lose money because they believed a "winner" that was never really winning — it just got lucky. That gap between what the data shows and what's actually true is exactly what the null hypothesis is built to catch. If you've ever shipped a variant because it "felt significant" and watched revenue quietly slide the following quarter, this is the concept you skipped.
TL;DR
- The null hypothesis assumes your variant has no real effect on conversion — any difference you see is random noise until proven otherwise.
- Standard significance threshold is 95% confidence (p < 0.05), meaning a 1-in-20 chance you're wrong even when a test "passes."
- ABWatcher detected a live hero copy test at AB Tasty on 8 Sep 2026 — headline ending in "personalization" vs. "conversions."
- Magic Spoon ran a full hero teardown-and-rebuild test with placeholder "Testing Testing Testing" copy, detected 29 May 2026.
- Attio tested pricing itself — $29 vs $35 for its Plus plan — detected 8 Sep 2026, proving even price points get null-hypothesis treatment.
- Calling a winner before reaching statistical significance is the single most common cause of false positives in landing page tests.
What is the null hypothesis in an A/B test?
The null hypothesis (H0) is the default assumption that your variant performs no differently than your control — full stop. Every A/B test starts from that skeptical baseline: the new hero, the new price, the new form image did nothing. Your job isn't to prove your idea is good. It's to gather enough evidence to reject the assumption that it did nothing. That reframe matters because it changes how you read early results — a 2% lift on day two isn't a win, it's a data point that hasn't yet cleared the bar to reject H0.
The alternative hypothesis (H1) is simply "there is a real difference." Statistical testing exists to tell you, with a stated confidence level, whether your observed difference is more likely explained by H1 or by ordinary sampling noise under H0. When people say a test is "statistically significant," they mean the data made H0 unlikely enough — typically less than a 5% chance — that they're comfortable rejecting it.
Why does the null hypothesis matter for landing pages specifically?
Landing pages have low, noisy baseline conversion rates, which makes false winners especially easy to manufacture by accident. A B2C landing page might convert at 2-4%. A B2B demo request page might convert at 1-2%. At those rates, sample sizes need to be large before random clustering of conversions stops looking like a trend. A weekend traffic spike, a paid campaign pause, or even a single enterprise deal closing can move your variant's conversion rate by a full percentage point with zero relation to the page change you shipped.
This is why structured A/B testing that isolates one variable matters so much — if you change the headline and the CTA button simultaneously, you can't attribute a lift to either one, and you definitely can't calculate a clean p-value against your null hypothesis. Every additional variable you test at once adds noise that makes rejecting H0 harder and slower.
Here's what that discipline looks like when brands actually run it live:
| Brand | Surface tested | What varied | Detected |
|---|---|---|---|
| Magic Spoon | Hero section | Full imagery hero vs. placeholder card with nutrition callouts | 29 May 2026 |
| AB Tasty | Hero headline | "...personalization" vs. "...conversions" | 8 Sep 2026 |
| Beis | Email capture popup + footer form | Lifestyle image + CTA copy ("SIGN UP FOR TEXTS" vs. "SIGN UP NOW") | 8 Sep 2026 |
| Attio | Pricing page | Plus $29 vs $35, Pro $69 vs $79, differing feature rows | 8 Sep 2026 |
| Ally | Hero + nav | Headline, image, CTA, and nav link presence | 8 Sep 2026 |
Notice the pattern: each of these isolates a defined surface. Attio didn't touch its hero while testing price. Beis kept the offer (20% off) constant and varied only the image and CTA copy. That's textbook single-variable isolation — it's what lets each brand cleanly attribute any conversion delta to one specific change instead of guessing.
How do you actually reject the null hypothesis without a stats degree?
You need three things: a large enough sample, a pre-set significance threshold, and the patience to wait for both. Most CRO tools calculate this for you, but understanding the mechanics protects you from misreading the dashboard. The standard threshold is 95% confidence, corresponding to a p-value under 0.05 — meaning if you ran this exact test 20 times with no real difference between variants, you'd expect to see a result this extreme by chance only once. That's still a real risk, which is why teams testing many things at once (five headlines, ten CTAs) need to correct their threshold or accept a higher false-positive rate across the whole batch.
Sample size calculators exist for a reason: they tell you, given your baseline conversion rate and the minimum lift you care about detecting, how many visitors per variant you need before checking results means anything. Ignore this step and you're vulnerable to "peeking" — checking results daily and stopping the moment they look favorable. Peeking is the single biggest driver of false winners because significance calculations assume a fixed sample size decided in advance, not a moving target you stop the instant it flatters you.
The Ally hero test is a useful mental model here. It didn't just swap a headline — it changed the hero image, CTA label, and even removed a "Credit Cards" nav link in one variant. That's a lot of simultaneous variables for one test, which means Ally's team likely treated this as a holistic page-experience test rather than a single-variable isolation test — a valid strategy, but one that requires much larger samples to reach significance because you're testing a bundle of changes, not one lever.
What happens when teams ship winners that never really won?
You get a false positive: a variant declared a winner that was, statistically, indistinguishable from the control — and then real revenue gets built on top of a coin flip. This is the exact failure mode the null hypothesis protects against, and it's expensive precisely because nobody notices right away. The test dashboard shows green, the team moves on, and three months later someone asks why conversion rate quietly drifted down after "the redesign."
Case studies on landing page restructuring show the flip side is just as real — one documented repositioning of trust signals and testimonials took a page from 3.91% to 6.38% conversion, a genuine 63% lift. The difference between that kind of durable win and a false positive that regresses later isn't luck; it's whether the team let the test run to a properly powered sample before calling it.
The math behind this connects directly to paid acquisition economics too. One modeled example shows a landing page test moving conversion rate from 4% to 4.8% — an 0.8 percentage point gain that lowers cost per lead by 16.7% at the same CPC. That's a meaningful number to report to finance. But it's only real if the 4.8% cleared statistical significance rather than reflecting one good week of traffic quality.
How should product and growth teams apply this on their next test?
Start every experiment by writing down the null hypothesis explicitly, not just the exploratory idea — literally state "this pricing change will have no effect on plan upgrades" before you launch, and require your data to overturn that specific sentence. Pre-register your minimum sample size and your significance threshold before you look at a single result. Resist checking results daily; if your tool supports sequential testing with proper corrections, use it, but don't eyeball a dashboard and call a winner on day three.
Layout, research shows, is one of the highest-leverage variables to test precisely because it affects the whole page's information hierarchy — which is exactly why bundled tests like Ally's hero-plus-nav change need bigger samples than single-element swaps like AB Tasty's headline word change.
Common questions
What's the difference between the null hypothesis and a control group?
The control group is the specific page or experience you're comparing against; the null hypothesis is the statistical claim that the control and variant produce no real difference in outcome. The control is a thing you show visitors. The null hypothesis is the assumption you're trying to statistically disprove using data from both groups.
How long should I run an A/B test before trusting the result?
Long enough to hit your pre-calculated sample size per variant, typically covering at least one full business cycle (often 1-2 weeks minimum) to average out day-of-week and traffic-source effects. Stopping early because results look good is the most common way teams accidentally accept a false positive.
Can a test be "directionally right" even if it's not statistically significant?
Directional trends can inform your next hypothesis, but they shouldn't drive a permanent rollout decision. Treat an underpowered result as a lead worth re-testing with a larger sample, not as evidence — this is precisely the discipline that separates the Attio-style clean pricing test from a rushed, noisy call.
Why do some brands test multiple elements at once, like Ally's hero and nav?
Bundled tests measure the effect of a whole redesigned experience rather than isolating one variable, which is useful when a team believes the page needs holistic change. The tradeoff is needing a much larger sample to reach significance, since you can't attribute the lift to any single element afterward.
Does the null hypothesis apply to non-visual tests, like pricing?
Yes — the same logic that governs a headline test applies to a price test. Attio's $29-vs-$35 Plus plan comparison starts from the same null assumption: that the price difference produces no meaningful change in upgrade rate, until the data says otherwise.
Put a one-line null hypothesis statement in every test brief this sprint — "this change will have no effect on [metric]" — and require your team to state the sample size needed to reject it before launch, not after you like the early numbers.
Read next
More from the ABWatcher blog
Nov 22, 2026
How to Calculate Confidence Intervals for Conversion Rate Lift
A practical guide to computing confidence intervals for A/B test conversion lift — frequentist and Bayesian methods, worked examples, and how to report ranges instead of fake certainty.
Nov 21, 2026
Why Your Conversion Rate Varies by Traffic Source (And What to Do About It)
A diagnostic guide to conversion rate variance across organic, paid, and referral traffic — how to separate real signal from source-quality noise before you spin up a new test.
Nov 20, 2026
Why Your A/B Test Winners Don't Replicate Across Traffic Sources
A test winner that holds for blended traffic often fails for a specific channel. Here's why, with real examples, and how to validate before you scale.
Nov 19, 2026
Session Recording for CRO: What to Watch When Users Bounce
A practical, intro-level guide to reading session replays for hidden friction and high-intent signals — plus how to turn what you see into testable hypotheses.
Nov 18, 2026
How to Build a Winning Landing Page Without A/B Testing
No traffic for statistical significance? Here's a design-led framework for building high-converting landing pages using proven patterns, not guesswork.
Nov 17, 2026
How to Find Your True Conversion Bottleneck (Not Just the Waterfall)
A step-by-step method for pinpointing the real cause of funnel drop-off using session recordings, heatmaps, and field-level friction data — before you waste a test cycle.
From the glossary
Vocabulary the article uses. See all terms →
- Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
- Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
- Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
- Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
- Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
- Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See more like this
ABWatcher catches A/B tests like this across 1,000+ brands.
Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.