CRO Glossary
Test Fatigue
The decline in experimentation quality or organizational buy-in that happens when teams run tests too frequently or without clear learning goals.
Test fatigue describes two related problems: audience fatigue, where repeat visitors are shown so many different experiments that their experience becomes inconsistent and their behavior no longer reflects a 'normal' visit, and organizational fatigue, where stakeholders lose enthusiasm for testing because too many experiments produce flat or inconclusive results without clear takeaways.
On the audience side, fatigue introduces noise: a returning shopper who saw three different checkout layouts in three visits isn't a clean data point for any of those experiments, and constant novelty or disruption can suppress conversion simply through inconsistency, unrelated to which layout is actually better.
On the organizational side, fatigue shows up as stakeholders asking 'why do we keep testing this' or deprioritizing experimentation altogether after a string of null results. This often stems from testing minor, low-impact changes just to keep a testing calendar full, rather than pursuing well-reasoned hypotheses tied to real user problems.
Teams manage this by capping how many concurrent experiments a single user can be exposed to, spacing out major UI changes, and — just as importantly — investing in a small number of high-quality tests with clear hypotheses and documented learnings (wins or losses) so the program keeps earning trust rather than draining it.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply test fatigue.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.