
Mailmodo Is A/B Testing Its Own Homepage Length — On Itself
Mailmodo, an email A/B testing company, is running a live Optimizely test on its own homepage — pitting a lean single-hero page against a long-form, feature-stacked one.
Aiko Tanaka
Design Director · · Updated
There's something delicious about catching an A/B testing company mid-A/B-test. Mailmodo's homepage at mailmodo.com/home-d-one/ is currently split into two very different experiences depending on which browser bucket you land in — and the gap isn't a button color or a headline tweak. It's the entire architecture of the page.
What We Actually Saw
Our vision-AI analyser flagged a same-run variance across five visits to the same URL. Visit 1, on Safari, rendered a short page: one hero section, a "Drive 3X engagement with actionable emails" module, and a lightweight footer. Done. Visits 2 through 5 — Chrome, Firefox, Edge — rendered a substantially longer page with the same hero up top, but then kept going: testimonials, a "Every email task, done in seconds with prompts" feature section, "Need a hand with setup?", "Email automations as easy as texting", an FAQ block, and more before the footer.
The cookie layer confirmed what the layout already suggested — Optimizely and EXP experiment keys were present only in the non-Safari visits, which is a clean signal of deliberate bucketing rather than a rendering bug. Safari visitors are seeing what looks like a control. Everyone else is getting the experiment variant. Confidence on this read sits at 92%.
Reading the Control: A Page Built for the First Scroll
The short variant is a bet on restraint. One hero, one supporting proof point, exit. There's no scroll depth to manage because there's barely any scroll. Everything a visitor needs to decide "is this relevant to me" lives above or just below the fold — no F-pattern decay to worry about because the page never asks the eye to travel that far.
This is the classic single-CTA landing page logic: fewer decision points, fewer places for attention to leak. If the hero copy and the "3X engagement" module are doing their job, the visitor either converts on the strength of that one clear promise or bounces. There's no room for a wandering FAQ section to introduce doubt that wasn't already there. It's a page designed to answer one question — what is this, and should I act — and then stop talking.
Reading the Variant: A Page Built for the Skeptic
The long version is a different theory of the visitor entirely. It assumes the person landing on this page isn't ready to act off a single hero claim — they need to be walked, in sequence, through proof, mechanism, and reassurance before the ask feels safe.
The ordering matters here. Testimonials come early, right after the hero, doing the social-proof job before the visitor has had time to generate objections. Then the page moves into mechanism — "Every email task, done in seconds with prompts" — which answers how, not just that. Then it addresses friction directly with "Need a hand with setup?", which is a smart placement: it's the exact moment a feature-curious visitor starts wondering about implementation cost, and the page meets that hesitation head-on rather than letting it fester into a bounce. "Email automations as easy as texting" reads like a closing reassurance beat — plain-language, low-intimidation — right before the FAQ mops up whatever's left.
Structurally, this is a page using progressive disclosure as a trust-building ramp. Each section is a rebuttal to the objection the previous section would have raised in a skeptical mind. That's a legitimate long-form playbook, but it only works if the visual hierarchy through each section is disciplined — if testimonials look like an afterthought, or the FAQ has no real contrast against a wall of body copy, the added length becomes added noise instead of added trust.
The Real Question: Fold Weight vs. Fold Count
What I find more interesting than "long vs short" is what each variant is asking of the fold. The short page puts all its fold weight on the hero — every pixel above that first scroll has to carry the entire value proposition, because there's nothing behind it to catch a visitor who almost clicks but hesitates. That's high pressure on a small amount of real estate, and it demands a hero with strong contrast, a legible hierarchy between headline and subhead, and a CTA that doesn't have to compete with anything else on screen.
The long page distributes that pressure across the scroll. The hero doesn't have to close the deal alone — it just has to earn the next scroll. That's a lower-stakes hero, which usually means the design team can afford to make it feel calmer, more editorial, less shouty. But it shifts the burden downstream: each section transition needs enough visual separation — spacing, background shifts, iconography changes — that the page doesn't blur into homogeneous scroll-fatigue by the time it reaches the FAQ.
Neither approach is inherently correct. It's genuinely a hypothesis worth testing, and it's exactly the kind of test most CRO teams under-run because it touches so many sections at once — it's not a one-variable test, it's a whole-page redesign wearing an A/B test's clothing. That's harder to ship, harder to instrument, and harder to read cleanly when it wins or loses, which is probably why it's rare to catch companies testing at this scope rather than button-color tweaks.
Why This Test Is Structurally Risky to Read
The tradeoff with a page-depth test this broad is attribution. If the long variant wins, is it the testimonials, the setup-friction section, the FAQ, or the cumulative effect of all of them together? Mailmodo's own A/B testing guide flags this exact trap — bundling too many changes into one variant makes the "why" unreadable even when the "what" (a lift or a drop) is statistically solid. They'll know which page performed better. They may not know which section did the work, unless there's a second layer of testing planned to pull the winning variant apart, section by section.
Takeaway for Your Sprint
Before you run a full page-depth test like this one, decide up front whether you're testing a hypothesis or testing a redesign. If it's a redesign, budget for a follow-up round of section-level tests on the winner — don't let "long page won" be the final answer when the real answer is buried three sections down. Test the whole page if you must, but plan to disassemble the winner immediately after.
Read next
More from the ABWatcher blog
Nov 22, 2026
Snake River Farms Tests a $25-Off Popup on Its Ad Landing Page
We caught Snake River Farms split-testing an email-capture form and $25-off modal on its landing page. Here's the math on what it would take to call a winner.
Nov 21, 2026
Snake River Farms Tests Mid-Page Email Capture on Its Ad Landing Page
SRF is A/B testing a mid-page lead-capture block and a $25-off popup against a simpler hero-only promo banner on its paid landing page, detected 2024 week-over-week.
Nov 20, 2026
Attio Is Testing a New Tier, Higher Prices, and a New Pitch
Attio is running a live pricing page test with a new $1,000 "Ultra" tier, higher Plus/Pro prices, and a new headline — here's what we caught and what it means.
Nov 19, 2026
Found Quietly Tested a 25% Price Cut on Its Pricing Page
We caught Found running a live pricing A/B test with an annual billing toggle that drops Found Plus from $35 to $26.25/mo. Here's what changed and why it matters.
Nov 8, 2026
Astro Proves "Zero Lock-In" With a Live Code Demo, Not a Claim
Astro's homepage replaces a trust badge row with a working, clickable framework demo — here's the hierarchy and interaction design behind it, detected 60% rare on astro.build.
Nov 7, 2026
Basecamp's Big Numbers: Why Hyper-Precise Stats Beat Vague Social Proof
Basecamp's homepage swaps "trusted by thousands" for 1.7 billion to-dos and 5 petabytes of data. We break down why precision beats vagueness as social proof — and how to test it.
From the glossary
Vocabulary the article uses. See all terms →
- Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
- Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
- Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
- Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
- Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
- Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See more like this
ABWatcher catches A/B tests like this across 1,000+ brands.
Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.