CRO Glossary
P-Value
The probability of observing a result as extreme as the one measured, assuming the null hypothesis is true.
A p-value tells you how surprising your experiment's data would be if there were actually no difference between variants. A small p-value (conventionally under 0.05) suggests the observed difference is unlikely to be due to chance alone, which is why teams use it as a threshold for calling a test 'significant.' It does not, however, tell you the probability that your variant is actually better, nor the size of the effect — it's purely a statement about how compatible the data is with a hypothesis of no effect.
Misreading p-values is one of the most common mistakes in experimentation. A p-value of 0.049 is not meaningfully different from 0.051, yet many teams treat 0.05 as a hard cliff between 'ship it' and 'kill it.' Combined with peeking (checking results repeatedly before a test finishes), this can produce false positives far more often than the nominal 5% rate suggests.
For example, if a checkout redesign produces a p-value of 0.03 when compared to control, that means there's roughly a 3% chance of seeing a difference this large (or larger) if the redesign truly had no effect. It's evidence worth acting on, but it should be considered alongside effect size, confidence intervals, and business context rather than in isolation.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply p-value.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.