
Snake River Farms Is Testing Whether to Kill Its Email Popup Entirely
Snake River Farms is running a high-risk full-element A/B test on its $25-off landing page — one variant strips the email-gate popup and exit-intent modal completely. Here's the math on what that bet costs or earns.
Sam Lee
Data Analyst · · Updated
TL;DR
- Detected 25 off landing page test at snakeriverfarms.com/pages/ad-top-standard, confidence 75%.
- Control shows a mid-page "$25 off sitewide" email capture block plus an exit-intent popup.
- Variant (caught on a second Chrome visit) shows neither — footer testimonials and the cookie banner load directly instead.
- Two other Chrome visits (5 and 6) matched control, confirming this is cookie/bucket-based, not browser rendering noise.
- Detection method:
vendor_sdk_active, meaning an A/B testing SDK was live on page, not just a visual diff. - This is a full element presence/absence test — the highest-risk test category in our taxonomy, per ABWatcher's writeup.
What did Snake River Farms actually change?
The variant doesn't tweak copy or button color — it deletes the entire lead-gen apparatus from the page. In the control experience, visitors hit a mid-page block prompting email signup for "$25 off sitewide," and if they move to exit the tab, an exit-intent popup fires as a second capture attempt. In the variant we caught, both are gone. The page flows straight from content into footer testimonials and the standard cookie consent banner. No fallback lighter-weight form, no delayed popup, no scroll-triggered nudge — just absence. That's the detail worth sitting with: this isn't a form-field-count test (4 fields vs. 2) or a CTA color swap. It's binary. The lead-gen surface exists, or it doesn't.
Why does the two-Chrome-visits detail matter?
It's the evidence that separates a real experiment from a rendering glitch, and it's the reason our confidence sits at 75% rather than lower. Any single divergent screenshot could be explained by a stale cache, a cookie consent state, or a flaky script load. But we saw three same-engine visits: two (visits 5 and 6) rendered the standard double-capture control, and one rendered the stripped-down variant. Same Chrome, same rendering engine, same device class — different experience. That pattern is the signature of a cookie- or session-bucketed split test, not noise. Combined with an active testing SDK on the page (vendor_sdk_active), this clears our bar for "live experiment" rather than "inconsistent build."
What's the actual hypothesis, and what's the power problem?
Snake River Farms is almost certainly testing incremental email capture against uninterrupted browsing — but a full-removal test like this has a much steeper evidentiary bar than most teams budget for. Popups and mid-page capture blocks are usually justified by list growth, not immediate purchase lift, so the real question is which metric wins: email signups (a leading indicator) or add-to-cart/purchase rate (the metric that pays the bills). If SRF is watching purchase conversion as the primary metric, they need a baseline conversion rate and enough daily sessions on this specific ad-landing URL to detect a realistic effect size.
Back-of-envelope: if this landing page runs at a 3% baseline conversion rate and they want to detect a 15% relative lift (0.45pp absolute) at 80% power and 95% confidence, that's roughly 14,000–16,000 sessions per arm — call it 30,000 total. If this is a single paid-ad landing page rather than sitewide traffic, that sample size can take weeks, and a two-week read is a classic way to mistake noise for a winner. Convert's writeup on tracking form submissions makes the same point about form tests generally — the "reduce fields, get more conversions" heuristic is directionally true on average but wildly variable by traffic source and intent, which is exactly the segmentation risk here.
What's the risk profile of a full-element test like this?
High, on both sides of the ledger, which is why it belongs in the "high risk" tier rather than treated like a routine CTA test.
| Test type | What's varied | Risk level | Is SRF running it? |
|---|---|---|---|
| Copy/headline test | Words only | Low | No |
| Button color/CTA test | Visual only | Low | No |
| Form field count test | Fields shown/hidden | Medium | No |
| Full element presence/absence | Entire form + popup exist or don't | High | Yes |
Removing the popup could lift on-page engagement and reduce bounce from annoyed visitors — popup fatigue is real and well documented. But it also removes SRF's primary email-capture funnel for cold traffic landing from a paid ad. If the $25-off popup currently drives, say, 8% of sessions into the list and 20% of those eventually convert at a $150 AOV, killing that popup sitewide would need the "clean" variant to make up the difference through some other channel — repeat visits, retargeting, or higher on-page conversion — or it's a net revenue loser even if it "feels" better. On a landing page pushing meaningful ad spend, a 1-point swing in email capture rate is not a rounding error; it compounds through retargeting and lifecycle revenue for months.
What should growth teams borrow from this?
Test type risk should scale your sample size and your metric hierarchy, not just your patience. A copy test can ship on a vibe and a week of data. A full-element removal test — popup gone, form gone — needs a pre-registered primary metric (purchase conversion, not just email opt-in), a power calculation before launch, and ideally a segment cut by traffic source, since paid-ad landing page visitors behave differently from organic or email-list visitors, a point Digital Applied's 2026 CRO guide makes directly about traffic-source segmentation outperforming one-size-fits-all page variants. This sprint: audit any test on your roadmap that removes rather than tweaks an element, and make sure you have both the sample size and the downstream revenue metric — not just the top-of-funnel one — before you call a winner.
Common questions
Is Snake River Farms definitely running an A/B test here?
Very likely. We detected an active testing SDK (vendor_sdk_active) on the page, and three same-browser visits split into two distinct experiences rather than one consistent render, which is the signature of cookie-bucketed variant assignment rather than a rendering bug. Analyst confidence is 75%.
Why does removing a popup count as "high risk" compared to a color change?
A color change affects visual preference only and rarely moves conversion by more than a percentage point or two. Removing the entire email-capture form and exit-intent popup changes the funnel structure itself — it can shift both email list growth and purchase conversion, sometimes in opposite directions, which makes the test's outcome harder to interpret cleanly.
How much traffic would SRF need to trust a result here?
Assuming roughly a 3% baseline landing-page conversion rate and a target of detecting a 15% relative lift, that's approximately 14,000–16,000 sessions per variant, or about 30,000 total, at 80% power and 95% confidence. Lower-traffic paid landing pages can take several weeks to hit that.
Could this just be novelty effect rather than a real preference shift?
Yes — that's a standard risk with any test that removes friction. Visitors may engage more in week one simply because the page feels less interruptive, then regress toward baseline as the novelty fades. A proper read requires a stable trend over at least 2-3 weeks, not just an early lift.
What metric should determine the winner, email signups or purchases?
Purchase conversion and revenue per session should be primary, with email capture rate tracked as a secondary/diagnostic metric. A variant that wins on email signups but loses on immediate revenue may still be worth it long-term, but that requires modeling downstream email-driven revenue, not just the top-of-funnel form-fill number.
Read next
More from the ABWatcher blog
Nov 22, 2026
Snake River Farms Tests a $25-Off Popup on Its Ad Landing Page
We caught Snake River Farms split-testing an email-capture form and $25-off modal on its landing page. Here's the math on what it would take to call a winner.
Nov 21, 2026
Snake River Farms Tests Mid-Page Email Capture on Its Ad Landing Page
SRF is A/B testing a mid-page lead-capture block and a $25-off popup against a simpler hero-only promo banner on its paid landing page, detected 2024 week-over-week.
Nov 20, 2026
Attio Is Testing a New Tier, Higher Prices, and a New Pitch
Attio is running a live pricing page test with a new $1,000 "Ultra" tier, higher Plus/Pro prices, and a new headline — here's what we caught and what it means.
Nov 19, 2026
Found Quietly Tested a 25% Price Cut on Its Pricing Page
We caught Found running a live pricing A/B test with an annual billing toggle that drops Found Plus from $35 to $26.25/mo. Here's what changed and why it matters.
Nov 8, 2026
Astro Proves "Zero Lock-In" With a Live Code Demo, Not a Claim
Astro's homepage replaces a trust badge row with a working, clickable framework demo — here's the hierarchy and interaction design behind it, detected 60% rare on astro.build.
Nov 7, 2026
Basecamp's Big Numbers: Why Hyper-Precise Stats Beat Vague Social Proof
Basecamp's homepage swaps "trusted by thousands" for 1.7 billion to-dos and 5 petabytes of data. We break down why precision beats vagueness as social proof — and how to test it.
From the glossary
Vocabulary the article uses. See all terms →
- Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
- Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
- Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
- Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
- Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
- Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See more like this
ABWatcher catches A/B tests like this across 1,000+ brands.
Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.