CRO Glossary
P-Value
The probability of observing a result as extreme as the one measured, assuming the null hypothesis is true.
A p-value tells you how surprising your experiment's data would be if there were actually no difference between variants. A small p-value (conventionally under 0.05) suggests the observed difference is unlikely to be due to chance alone, which is why teams use it as a threshold for calling a test 'significant.' It does not, however, tell you the probability that your variant is actually better, nor the size of the effect — it's purely a statement about how compatible the data is with a hypothesis of no effect.
Misreading p-values is one of the most common mistakes in experimentation. A p-value of 0.049 is not meaningfully different from 0.051, yet many teams treat 0.05 as a hard cliff between 'ship it' and 'kill it.' Combined with peeking (checking results repeatedly before a test finishes), this can produce false positives far more often than the nominal 5% rate suggests.
For example, if a checkout redesign produces a p-value of 0.03 when compared to control, that means there's roughly a 3% chance of seeing a difference this large (or larger) if the redesign truly had no effect. It's evidence worth acting on, but it should be considered alongside effect size, confidence intervals, and business context rather than in isolation.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply p-value.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.