Snake River Farms Swaps Testimonials for an Email Gate — Is It Working?

The ABWatcher blog

Snake River Farms Swaps Testimonials for an Email Gate — Is It Working?

Snake River Farms is running a same-page A/B test that replaces testimonials with an email-gated $25-off form and exit-intent popup. Here's the math on whether that trade is likely a net win.

Sam Lee

Data Analyst · · Updated

TL;DR

  • Brand: Snake River Farms (snakeriverfarms.com), premium meat DTC ecommerce.
  • Page: landing page at /pages/ad-top-equal, detected via same-run variance across 7 visits.
  • Variant A (Safari, visit 1): full page with a testimonials section, no popup.
  • Variant B (Firefox, Edge, Chrome x2, Firefox — visits 3-7): testimonials replaced with an email-gated "Sign up for access / $25 off sitewide" form, plus an exit-intent discount popup.
  • Detection method: same_run_variance — the split held across three distinct browser engines (WebKit, Gecko, Chromium), ruling out a rendering bug.
  • Analyst confidence: 80%, logged as a genuine live experiment, not a caching artifact.

What did the variance actually show?

Across 7 recorded visits, one browser engine saw testimonials and six saw an email gate plus exit popup instead — a pattern too consistent to be noise. The tell here is dispersion across engines, not just across visits: Safari rendered the "control" (testimonials), while Firefox, Edge, and Chrome — two separate visits each on Chrome and Firefox — all rendered the "treatment" (email gate + exit-intent popup). If this were a caching or rendering glitch, you'd expect it to cluster by engine or by session recency, not to split cleanly into two consistent buckets across six of seven visits. That's the signature our same_run_variance method is built to catch — genuine bucket assignment, not artifact.

The magnitude of the change also argues for "real test": this isn't a button color swap, it's a full section replacement (social proof → gated capture form) plus the addition of an entirely new interaction (exit-intent popup). Conversion Sciences calls this a "Big Swing" — the kind of test teams run when they suspect the page's core value prop or funnel logic is underperforming, not when they're doing incremental polish.

Why would Snake River Farms trade testimonials for a gate?

The likely hypothesis is that email capture plus a $25 discount lever generates more downstream value than on-page social proof alone. Testimonials do work — they're a trust signal for a $150+ box of wagyu you can't touch before buying — but they only convert visitors in that session. An email-gated $25-off offer converts visitors into a list asset that can be remarketed for months. On a landing page explicitly named ad-top-equal (almost certainly a paid-traffic landing page, likely for an "equal" or top-of-funnel ad set), the marginal value of a captured email address for retargeting could easily exceed the marginal trust lift from testimonials, especially if paid CAC is already doing the trust-building work upstream via ad creative.

That said, this is a layout+element test bundled together (gate section and exit popup), which technically muddies attribution. As TruConversion notes, testing more than one variable at once turns a clean A/B into something closer to a multivariate test — if the variant wins, Snake River Farms won't cleanly know whether the lift came from the gate, the popup, or the combination.

What would a real read on this test require?

You'd need enough sessions per arm to detect a plausible ecommerce lift with reasonable power, and nothing in the detection tells us they have it. For a landing page conversion rate baseline of, say, 3%, detecting a 15% relative lift (0.45pp absolute) at 80% power and 95% confidence needs roughly 14,000–16,000 sessions per arm — not unusual for a paid landing page, but worth stating plainly rather than assuming. Contentful's guidance on layout testing is right that layout changes "can greatly affect how visitors interact," but effect size claims without a sample size and interval are just anecdotes with better formatting.

There's also a real risk of a novelty effect running the wrong direction: exit-intent popups reliably lift email capture in the short run (that's well documented), but they can suppress the browse-to-purchase path for return visitors who get annoyed on repeat sessions. If Snake River Farms is only measuring immediate opt-in rate, they may be logging a "win" that's actually a shift from purchase-intent traffic to lower-intent list traffic. A rough sanity check: if the gate variant lifts email signups by 25% but purchase conversion on that same page drops even 2 points, on a landing page doing $2–5M in attributed annual revenue, that 2-point drop is a $40K–$100K annualized hit that a signups-only dashboard would never surface.

How should you read this if you run similar pages?

This table shows how the two variants differ and where the ambiguity sits:

Element Variant A (Safari) Variant B (Firefox/Edge/Chrome) Confound risk
Testimonials section Present Removed High — removes trust signal entirely
Email capture form Absent "$25 off sitewide," gated Primary test variable
Exit-intent popup Absent Present, discount-based Bundled second variable
Browser engines seen WebKit (1 visit) Gecko + Chromium (6 visits) Rules out rendering bug

The practical fix for anyone replicating this test on their own roadmap: split the gate and the popup into separate cells (A: testimonials, B: gate only, C: testimonials + popup, D: gate + popup) so you can isolate which lever is doing the work, and pre-register your primary metric as revenue-per-visitor, not signup rate, before you launch. Per WOLF Financial's benchmark table, social proof positioning changes typically move conversion 8–20%, while form-field changes move it 20–45% — meaning the email-gate swap likely has more raw conversion leverage than the testimonial removal costs, but that's a population-level benchmark, not a guarantee for a niche premium-meat audience with a materially different trust threshold.

Common questions

Is this confirmed to be a real A/B test and not a bug?

Yes, with 80% analyst confidence. The same split (testimonials vs. gate+popup) appeared consistently across three different browser engines — WebKit, Gecko, and Chromium — across 6 of 7 visits, which is the pattern same_run_variance detection is designed to flag as genuine bucketing rather than a caching or rendering artifact.

Why would removing testimonials be worth the risk?

If the page's traffic source (paid ads, given the ad-top-equal URL slug) already carries brand trust, on-page testimonials may add less marginal conversion than a $25-off email capture that extends the relationship past one session. It's a bet that list-building beats immediate on-page social proof for this specific funnel stage.

What sample size would Snake River Farms need to trust a result here?

To detect a 15% relative lift on a 3% baseline conversion rate at 95% confidence and 80% power, you need roughly 14,000–16,000 sessions per variant. Without knowing this page's traffic volume, we can't confirm they've reached that threshold, so any early "winner" should be treated as directional only.

Could the exit-intent popup be hurting more than it helps?

Possibly. Exit popups reliably boost short-term email opt-ins but can suppress purchase conversion for repeat visitors who find them intrusive. If Snake River Farms is optimizing for signup rate alone rather than revenue-per-visitor, they risk shipping a variant that grows the list while quietly shrinking sales on the same page.

What should teams running a similar test do differently?

Split the gate and the popup into separate test cells instead of bundling them, so you can attribute lift to a single variable. Also pre-register revenue-per-visitor as the primary metric, not signup count, since email capture and purchase conversion can move in opposite directions on the same variant.

Takeaway for your next sprint

If you're planning a similar swap — trust content out, email gate in — don't ship it as one bundled variant. Run testimonials-vs-gate and popup-on-vs-off as separate cells, hold revenue-per-visitor as the primary metric rather than opt-in rate, and don't call a winner until you've cleared the sample size math for your actual baseline conversion rate. A 25% lift in signups on a few hundred sessions per arm isn't a result — it's a hypothesis waiting for power.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.