CRO Glossary
Analysis Paralysis
A stall in decision-making caused by over-analyzing experiment data or waiting for more certainty than is practical to obtain.
Analysis paralysis happens when a team has enough data to make a reasonable call but keeps digging for more segments, more confidence, or more edge cases instead of shipping a decision. It's the opposite failure mode from peeking too early — here the risk is moving too slowly, not too fast, and it quietly kills experiment velocity even when the testing program itself is technically sound.
This matters in CRO because every week spent re-slicing an already-concluded test is a week not spent running the next one. Programs judged by throughput (how many validated learnings ship per quarter) suffer badly when stakeholders demand one more cut of the data 'just to be sure,' especially for results that are directionally clear but not dramatic.
A common trigger is a result that's statistically significant but smaller than hoped, prompting endless segment-by-segment investigation looking for a story that justifies the effort invested. Setting decision criteria and kill criteria before a test launches — what result, at what confidence, will trigger a ship/no-ship call — is the standard antidote, because it removes the temptation to negotiate with the data after the fact.
Example: a pricing page test shows a 3% lift with solid significance, but the team spends five more weeks segmenting by browser, geography, and referral source looking for a bigger number, rather than shipping the win and moving to the next test in the backlog.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply analysis paralysis.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.