Snake River Farms Tests a $25-Off Popup on Its Ad Landing Page

The ABWatcher blog

Snake River Farms Tests a $25-Off Popup on Its Ad Landing Page

We caught Snake River Farms split-testing an email-capture form and $25-off modal on its landing page. Here's the math on what it would take to call a winner.

Sam Lee

Data Analyst · · Updated

TL;DR

  • Brand: Snake River Farms, tested on its paid landing page snakeriverfarms.com/pages/ad-landing.
  • Variant A adds a "$25 OFF sign up for access" email-capture form plus a below-the-fold discount modal; Variant B omits both.
  • Detection method: same_run_variance — identical browser engines (Chrome visit 2 vs. visit 5, Firefox visit 7 vs. visit 3) rendered different versions, ruling out a rendering bug.
  • Analyst confidence on this being a genuine live A/B test: 85%.
  • Surface under test: form / lead-capture mechanics, not pricing or layout.
  • No stated end date or sample size disclosed by the brand — test status is inferred from live traffic snapshots only.

What exactly did we catch on the Snake River Farms landing page?

Our vision-AI analyser flagged two materially different experiences served from the same URL on the same day. One variant shows an email-capture block promising "$25 OFF sign up for access," reinforced by a second $25-off popup modal that fires below the fold. The other variant strips both elements — no form, no modal, just the base landing page. Because this is an ad-landing page (likely fed by paid social or search), the stakes per visitor are higher than on organic blog traffic: every session already cost money to acquire.

The tell that this is a deliberate experiment and not a caching glitch is the cross-browser pattern. Same_run_variance detection means we saw Chrome serve both versions across different visits in the same monitoring run, and Firefox do the same. If this were a CDN edge-cache inconsistency or a stale service worker, you'd expect it to correlate with browser engine or device, not to flip randomly within the same engine. That rules out the two most common false positives in our detection pipeline and is why we're logging this at 85% confidence rather than treating it as noise.

Why would Snake River Farms test an email popup on a paid landing page?

The likely hypothesis is a classic lead-capture-vs-friction tradeoff: does a $25-off form plus modal lift opt-in and downstream conversion, or does it add friction and tank bounce rate on expensive paid traffic. On an ad-landing page, the brand is essentially deciding between two monetization paths — capture the email now at a discount cost, or let the visitor browse straight to product/checkout with a cleaner page. Both are legitimate growth levers; the question is which one pays back the $25 discount plus any incremental bounce.

Worth noting: a modal that fires below the fold is lower-intrusion than an exit-intent or immediate-load popup, which suggests the team is already hedging against a UX complaint — likely informed by prior tests or support tickets. That's a sensible middle path, but it also means the lift, if any, is probably going to be modest rather than dramatic, because the intervention itself is soft-touch.

How big a sample would Snake River Farms need to trust this result?

Back-of-envelope: if baseline landing-page conversion (purchase or qualified lead) sits around 3%, detecting a 15% relative lift (3.0% → 3.45%) at 80% power and 95% confidence requires roughly 14,000–15,000 visitors per arm — about 28,000–30,000 total. If the true effect is smaller, say a 5% relative lift, the required sample roughly triples to ~130,000 per arm. Paid landing pages rarely get there fast unless ad spend is substantial, so the real risk here is a team calling a winner at week two off a noisy top-line number.

The other thing to check before declaring victory: email capture rate and purchase conversion rate are two different metrics, and a form can win on the former while losing on the latter. If Variant A lifts email signups 20% but net landing-page conversion is flat or down 2%, that's not a win, that's a cost center — you're paying $25 off margin to collect an email with no incremental revenue attached. Contentful's framework on A/B test duration is useful here: run long enough for statistical significance, but don't let "long enough" become "until the number I like shows up."

What does the test matrix look like so far?

Here's what we've observed in the live monitoring snapshots used to flag this test:

Browser Visit # Variant served Form + modal present?
Chrome 2 A Yes
Chrome 5 B No
Firefox 7 A Yes
Firefox 3 B No

Four data points isn't a test result — it's a detection. The split pattern across identical engines is what gives us confidence this is deliberate traffic allocation rather than a caching artifact, but it tells us nothing about which variant is winning. We don't have visibility into Snake River Farms' internal conversion dashboard, so any lift estimate right now would be speculation dressed up as data.

Is a form-plus-popup test actually a novelty-effect risk?

Yes, and it's worth flagging before anyone on the Snake River Farms team (or a reader running a similar test) calls this early. Popups and discount offers tend to show an initial engagement bump purely because they're new and because early traffic often skews toward more engaged, bookmark-and-return visitors. Kameleoon's guide to A/B testing rightly frames every test as a learning opportunity about customer behavior over time — the operative phrase being "over time." A two-week test window on a seasonal meat/ecommerce brand is also vulnerable to promotional calendar effects: if this ran across a holiday push, the $25-off variant could look artificially strong or weak depending on what competing offers were in-market.

Common questions

Is this confirmed to be an active A/B test?

We're at 85% analyst confidence based on same_run_variance detection — identical browser engines serving different variants across visits, which rules out rendering bugs or caching as the cause. It has not been confirmed by Snake River Farms directly, so treat it as high-confidence inference, not an official disclosure.

What metric should determine the winner here?

Net landing-page conversion to purchase (or qualified lead-to-sale rate), not raw email opt-in rate. A form can inflate signups while cannibalizing checkout conversion if the $25 discount trains visitors to wait for an offer before buying, so the discount's margin cost needs to be netted against incremental revenue.

How long should a test like this run before trusting the result?

Long enough to hit your required sample size at your chosen minimum detectable effect — commonly 2-4 weeks for a landing page with decent ad spend, but it depends entirely on baseline traffic and conversion rate. Stopping early on a favorable peek is the single most common way teams manufacture false wins.

Could the popup actually hurt this brand's conversion rate?

Yes — a below-the-fold modal is lower-friction than an immediate popup, but any added step before checkout risks raising bounce on paid traffic, where every visitor already cost money to acquire. If bounce rate rises even 2-3 percentage points, it can offset the value of the extra emails collected.

Takeaway for your own roadmap

If you're planning a similar email-capture-plus-discount test this sprint, pre-register your required sample size and your primary metric (net conversion, not opt-in rate) before you launch — and build in a novelty-effect check by comparing week-one lift to week-three lift on the same variant. If the lift decays, you were measuring curiosity, not preference.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.