CRO Glossary
Bandit Test
An experimentation approach that dynamically shifts traffic toward better-performing variants while the test is still running.
Unlike a classic A/B test, which holds traffic allocation fixed until a predetermined end point, a multi-armed bandit test continuously reallocates traffic in favor of whichever variant is currently performing best. The algorithm balances 'exploration' (still testing all variants to gather data) against 'exploitation' (sending more users to the apparent winner) to minimize the cost of showing an inferior variant to users.
Bandits are well suited to situations where the cost of a bad variant is high and the test doesn't need to produce a clean, publishable, statistically airtight comparison — for example, optimizing a homepage banner or headline during a short-lived marketing campaign or news event. They're generally a poor fit for situations requiring rigorous confidence in a specific effect size, since the constantly shifting allocation makes the resulting statistics harder to interpret than a fixed-split test.
A common example is testing five headline variants for a time-sensitive promotional email; a bandit algorithm will quickly funnel most opens toward whichever headline is getting the best click-through rate, reducing the number of users who ever see an underperforming version.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply bandit test.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.