Multi-Armed Bandit

Unlike a traditional fixed-split A/B test, a multi-armed bandit continuously monitors variant performance and reallocates traffic in real time, sending more visitors to whichever variant currently looks best. The name comes from the 'one-armed bandit' slot machine problem: you're trying to balance exploring unproven options against exploiting the option that already looks like a winner.

Bandits are attractive when the cost of showing a losing variant is high, such as a homepage banner during a short flash sale, because they minimize the traffic sent to underperforming options over the test's lifetime. This is different from a classic bandit test setup used for long-running experiments, though the underlying algorithm (like Thompson Sampling or epsilon-greedy) is the same.

The tradeoff is statistical rigor: because traffic allocation changes based on early results, bandits are more prone to bias from early noise and don't produce the same clean confidence interval or p-value output as a fixed-allocation test. They're best suited for short-lived, high-stakes decisions rather than situations where you need a defensible, generalizable causal estimate for a permanent product change.