CRO Glossary
Peeking Problem
The inflated false-positive risk that comes from repeatedly checking test results before the planned sample size is reached.
The peeking problem happens when a team monitors an experiment's results continuously and stops the test as soon as it crosses a significance threshold, rather than waiting until a pre-determined sample size or duration is reached. Each peek is effectively another chance for random noise to look like a real effect, and with enough peeks, the true false-positive rate can climb from a nominal 5% to 20%, 30%, or higher.
This matters because it's an extremely common and intuitive mistake — dashboards update in real time, and it's natural to check in daily and call a winner the moment the numbers look good. The problem isn't checking the dashboard itself; it's making a stop/ship decision based on an interim peek rather than the pre-registered analysis plan.
The standard fixes are to either commit to a fixed sample size and analysis date decided before the test starts, or to use sequential testing methods designed explicitly to allow valid peeking. For example, a team that plans to run a test for two weeks but checks results daily should either resist stopping early, or use a sequential testing framework that adjusts thresholds to keep the overall error rate under control.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply peeking problem.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.