The ABWatcher blog

How to Run Your First A/B Test: A Step-by-Step Guide for Beginners

A beginner's guide to running your first A/B test — hypothesis, sample size, metrics, and real live-test examples from SignNow, Prismatic, Dropbox Sign, Upwork, and GetYourGuide.

Sam Lee

Data Analyst · · Updated

Most first A/B tests fail before a single visitor is bucketed — not because the idea was bad, but because nobody did the arithmetic on sample size, picked a metric that actually maps to revenue, or wrote down what "win" means before the data came in. That's an expensive mistake: a company running an underpowered test for two weeks and calling a 3% lift a "win" is usually looking at noise, not signal. This guide walks through the mechanics you need before you ship your first test, with live examples from tests ABWatcher has actually detected on production sites.

TL;DR

  • A test measuring a baseline 5% conversion rate needs roughly 12,000–25,000 visitors per arm to detect a 10% relative lift at 80% power — not a few hundred.
  • SignNow (signnow.com) is running a hero A/B test detected 8 Aug 2026, splitting single-CTA vs. two-path homepage layouts plus a client-logo bar.
  • Dropbox Sign (dropbox.com) is testing pricing-page layout depth — full FAQ/comparison sections vs. a stripped-down pricing table — detected 5 Aug 2026.
  • Upwork (upwork.com) split-tested an entirely different hero headline, image, and CTA pair ("Find talent" / "Browse jobs"), detected 25 Jun 2026.
  • Standard stopping rule: don't call a result before you hit your pre-calculated sample size, even if p < 0.05 shows up early — that's peeking, and it inflates false positive rates well above 5%.

What actually counts as an A/B test hypothesis?

A hypothesis is a specific, falsifiable prediction that names the change, the metric, and the expected direction — not a vague idea like "improve the hero."

Write it in the format: "Changing [element] from [A] to [B] will [increase/decrease] [metric] for [segment] because [mechanism]." That last clause — the mechanism — is what separates a testable hypothesis from a guess. Take the SignNow test detected 8 Aug 2026: one variant shows a single "Get your first document signed" hero with one upload CTA and no logo strip; the other shows a two-column "Send my document for signature" / "Sign my own document" split plus a client-logo bar (Walmart, ExxonMobil, Apple, Comcast, Facebook, FedEx) and a "Why choose airSlate SignNow" section. The implicit hypothesis there is that pre-segmenting intent (send vs. sign) plus social proof from recognizable logos reduces bounce for cold traffic that doesn't yet trust the brand. That's specific enough to measure and specific enough to be wrong.

Compare that to Upwork's test, detected 25 Jun 2026, where one hero variant read "Connecting businesses in need to freelancers who deliver" with a different hero image, a "Find talent" / "Browse jobs" CTA pair, and a simplified pricing section ("Clients only pay after hiring") with fewer feature bullets. That's not one change — it's five stacked changes (headline, image, CTA copy, search placeholder, pricing depth) running as a single variant. That's a legitimate strategy for a first directional test when traffic is scarce, but you can't attribute a win to any one element from it. If you want to know whether the headline or the pricing simplification drove the lift, you need a follow-up test that isolates them.

How do you calculate the sample size before launching?

You need the baseline conversion rate, the minimum detectable effect (MDE) you care about, and your desired statistical power — typically 80% — before you route a single visitor.

The formula most tools use under the hood (two-proportion z-test) simplifies to roughly:

n ≈ 16 × p(1-p) / δ²

where p is your baseline rate and δ is the absolute difference you want to detect. If your signup page converts at 5% and you want to detect a 10% relative lift (0.5 percentage points, δ = 0.005), that's:

n ≈ 16 × 0.05 × 0.95 / 0.000025 ≈ 30,400 per arm

That's roughly 61,000 total visitors for one test — and that assumes no segment splitting. If you're running a pricing-page test like the one detected at Dropbox Sign, and pricing pages typically see 5–20x less traffic than the homepage, you may need 3-4 weeks minimum just to reach power, even for a mid-market SaaS company. This is why Optibase's guide to A/B testing and Convertize's quick-start guide both flag "stopping too early" as the single most common beginner mistake — significance can appear and disappear multiple times before a test reaches its planned sample, and calling it on day 4 of a 21-day run is a coin flip dressed up as data.

Which page sections are worth testing first?

Hero sections and pricing pages carry the highest leverage-per-visitor because they sit closest to the conversion decision, which is why they're the most commonly detected test surfaces in the wild.

Below is a breakdown of the five live tests referenced in this guide, by surface and what was actually varied:

Company Surface Detected What varied
SignNow Hero 8 Aug 2026 Single-CTA hero vs. two-path hero + logo strip + trust section
Prismatic Hero subheadline 5 Aug 2026 "14-day free trial" vs. generic "your free trial" copy
Dropbox Sign Pricing layout 5 Aug 2026 Pricing-only view vs. + add-ons, FAQ, and comparison table
Upwork Hero + pricing 25 Jun 2026 Full headline/image/CTA/pricing bundle swap
GetYourGuide Content layout 13 Jun 2026 "Unforgettable travel experiences" vs. "Go beyond the guidebook" with different cards

Notice the pattern: four of five tests touch the hero or the first content block below it. That tracks with broader CRO research — Unbounce's step-by-step guide and Digital Applied's 2026 CRO guide both point to above-the-fold messaging and pricing-page structure as the two highest-ROI test surfaces for a reason: they're high-traffic, low-engineering-cost to swap, and directly upstream of the primary conversion event.

The Prismatic test is a useful beginner case because it isolates a single variable cleanly: one subheadline says "14-day free trial," the other says generic "your free trial" phrasing, detected 5 Aug 2026 across multiple browser engines (ruling out rendering noise, since Firefox appeared in both groups). That's the level of isolation a first test should aim for — one variable, one metric, one hypothesis. Specificity in a trial-length callout is a testable claim about reducing signup friction through concreteness, and it's cheap to build.

What metric should you actually track?

Track one primary metric tied directly to revenue or activation, and treat everything else as a secondary or guardrail metric you monitor but don't optimize for.

For a SaaS trial signup flow, that's usually trial starts or, better, trial-to-paid conversion if your sample size supports waiting that long. For an ecommerce pricing or checkout test like the Dropbox Sign layout test detected 5 Aug 2026 — which varies whether add-ons, FAQ, and comparison sections render — the primary metric should be checkout initiation or plan-select clicks, not pageviews or scroll depth. Scroll depth and time-on-page are vanity metrics in this context: they tell you engagement changed, not whether it converted to money.

Set guardrails too. If a hero redesign like GetYourGuide's — detected 13 Jun 2026, swapping "Unforgettable travel experiences" for "Go beyond the guidebook" with entirely different destination cards — lifts click-through on activity cards but tanks search-bar usage, you need to have declared search usage as a guardrail metric beforehand, or you'll rationalize the drop after the fact. Post-hoc metric selection is how teams talk themselves into shipping a net-negative change.

Run the power calculation with your guardrail metrics too, not just your primary — a guardrail with a smaller baseline rate (say, a 1.5% add-on attach rate versus a 12% checkout rate) needs a much larger sample to detect the same relative movement, and you don't want to be blind to a guardrail regression just because you under-powered it.

How long should you run a test, and when should you stop?

Run until you hit your pre-calculated sample size or a pre-committed time cap (commonly 2-4 weeks), whichever comes second — not the moment you see p < 0.05.

Two failure modes bracket this. First, novelty effects: a new layout gets an initial lift purely because repeat visitors notice something different, and that lift decays over 1-2 weeks as novelty wears off. If you stop a test at day 5 because it's "winning," you're likely measuring novelty, not preference. Second, weekly seasonality: B2B tools like Prismatic or SignNow see different visitor mix and intent on Tuesday versus Saturday, so a test needs to run at least one full week — ideally two full business cycles — to average that out. Conversion Sciences' 6-step process recommends anchoring your stop date to calendar weeks for exactly this reason.

Once you do stop, report the result with a confidence interval, not a point estimate. "Variant B lifted signups by 8%" is not a claim; "Variant B lifted signups by 8%, 95% CI [2%, 14%], n=58,000" is. If your interval crosses zero, you don't have a winner — you have a test that needs more traffic or a bigger effect to detect.

Common questions

How many visitors do I need before I can trust an A/B test result?

It depends on your baseline conversion rate and the minimum lift you want to detect, but as a rule of thumb, detecting a 10% relative lift on a 5% baseline conversion rate requires roughly 30,000 visitors per arm at 80% power. Smaller baseline rates or smaller target lifts require dramatically more traffic — run the calculation before launch, not after.

Can I test more than one element at once, like Upwork did?

Yes, but you sacrifice attribution. Upwork's test detected 25 Jun 2026 changed headline, image, CTA copy, and pricing depth together — useful for a fast directional read with limited traffic, but you won't know which element drove any lift. Follow up by isolating the strongest candidate in a second, single-variable test.

What's a novelty effect and how do I avoid being fooled by it?

A novelty effect is a temporary lift caused by visitors noticing change, not preferring it — it typically fades within 1-2 weeks. Guard against it by running tests a minimum of two full weeks and checking whether the lift is stable or declining across that window before calling a winner.

Should I test the hero section or the pricing page first?

Test whichever gets more qualified traffic relative to your required sample size. Hero sections (like those tested by SignNow and Prismatic) usually reach significance faster due to higher volume; pricing pages (like Dropbox Sign's layout test) carry higher intent per visitor but often need 3-4x longer to reach the same statistical power.

What if my test shows a "win" but the confidence interval is wide?

Don't ship it as a permanent change yet. A wide interval — say [1%, 19%] — means your point estimate is unreliable even if it excludes zero. Extend the test to narrow the interval, or treat it as a hypothesis for a confirmatory re-run before committing engineering resources to a permanent rollout.

Your next step this sprint

Before you write a line of test code, pick one page, one metric, and run the sample-size math with your actual baseline conversion rate — not a guess. If the required sample size means your test would take longer than 6 weeks at current traffic, you don't have a testing problem yet, you have a traffic problem, and your first optimization should go toward acquisition, not experimentation. If the math works, borrow the isolation discipline from Prismatic's single-variable subheadline test rather than the five-variable bundle Upwork shipped, ship it, and don't look at results until you hit your pre-committed sample size or two full weeks, whichever is longer.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.