CRO Glossary
A/A Test
A test that splits traffic between two identical experiences to validate that the testing system itself is working correctly.
An A/A test runs the same version of a page or feature against itself, splitting visitors into two (or more) groups exactly as you would in a real experiment, but without changing anything they see. Because both groups are identical, you expect no meaningful difference in conversion rate or other metrics between them.
Teams run A/A tests to validate their experimentation infrastructure before trusting it with real decisions. If an A/A test reports a statistically significant difference between two identical groups more often than your chosen significance threshold would predict, that's a red flag: it can point to bugs in randomization, tracking, or analysis code, or to a sample ratio mismatch.
For example, a company migrating to a new experimentation platform might run an A/A test for two weeks on their checkout flow. If the tool reports a 'winner' with 95% confidence even though both variants are the same code, that suggests the tool's stats engine, event logging, or bucketing logic needs debugging before it's used for real A/B tests.
A/A tests are also useful for estimating baseline variance and noise in a metric, which can help calibrate expectations for future minimum detectable effect calculations.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply a/a test.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.