CRO Glossary
Multi-Armed Bandit
An algorithmic approach to testing that dynamically shifts traffic toward better-performing variants while the experiment is still running.
Unlike a traditional fixed-split A/B test, a multi-armed bandit continuously monitors variant performance and reallocates traffic in real time, sending more visitors to whichever variant currently looks best. The name comes from the 'one-armed bandit' slot machine problem: you're trying to balance exploring unproven options against exploiting the option that already looks like a winner.
Bandits are attractive when the cost of showing a losing variant is high, such as a homepage banner during a short flash sale, because they minimize the traffic sent to underperforming options over the test's lifetime. This is different from a classic bandit test setup used for long-running experiments, though the underlying algorithm (like Thompson Sampling or epsilon-greedy) is the same.
The tradeoff is statistical rigor: because traffic allocation changes based on early results, bandits are more prone to bias from early noise and don't produce the same clean confidence interval or p-value output as a fixed-allocation test. They're best suited for short-lived, high-stakes decisions rather than situations where you need a defensible, generalizable causal estimate for a permanent product change.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply multi-armed bandit.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.