CRO Glossary
Test Cannibalization Risk
The risk that a winning variant in one experiment shifts behavior in a way that erodes or distorts the results of a related experiment.
Test cannibalization risk refers to the possibility that an experiment's effect isn't a net gain for the business but rather a shift of behavior from one metric, channel, or page to another that is being measured separately. It's distinct from simple test contamination (users seeing overlapping variants); this is about the underlying user behavior itself moving around in a way that muddies interpretation.
This matters because teams often optimize individual pages or steps in isolation. A test that increases clicks on a homepage promotion might simply be pulling conversions forward from a different acquisition channel or a later-funnel test, rather than creating incremental revenue. If that offsetting effect isn't tracked, the team celebrates a lift that never shows up in top-line numbers.
A concrete example: an experiment that adds an aggressive discount banner to the homepage boosts homepage-to-cart conversion, but a concurrent email campaign test in the same period sees its conversion rate drop because the same users are converting via the banner instead of the email link. Each test looks fine in isolation, but combined they've just reallocated the same pool of buyers rather than growing overall conversion.
Guarding against this usually means looking at a shared north star or revenue metric across concurrent experiments, not just each test's local success metric, and being cautious about running multiple tests that target overlapping user intents at the same time.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply test cannibalization risk.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.