CRO Glossary
Test Fatigue
The decline in experimentation quality or organizational buy-in that happens when teams run tests too frequently or without clear learning goals.
Test fatigue describes two related problems: audience fatigue, where repeat visitors are shown so many different experiments that their experience becomes inconsistent and their behavior no longer reflects a 'normal' visit, and organizational fatigue, where stakeholders lose enthusiasm for testing because too many experiments produce flat or inconclusive results without clear takeaways.
On the audience side, fatigue introduces noise: a returning shopper who saw three different checkout layouts in three visits isn't a clean data point for any of those experiments, and constant novelty or disruption can suppress conversion simply through inconsistency, unrelated to which layout is actually better.
On the organizational side, fatigue shows up as stakeholders asking 'why do we keep testing this' or deprioritizing experimentation altogether after a string of null results. This often stems from testing minor, low-impact changes just to keep a testing calendar full, rather than pursuing well-reasoned hypotheses tied to real user problems.
Teams manage this by capping how many concurrent experiments a single user can be exposed to, spacing out major UI changes, and — just as importantly — investing in a small number of high-quality tests with clear hypotheses and documented learnings (wins or losses) so the program keeps earning trust rather than draining it.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply test fatigue.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.