CRO Glossary
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
Orthogonal test design is an approach to structuring multivariate experiments so that each factor being tested varies independently of the others across the full set of combinations shown to users. The goal is to be able to isolate the effect of each individual element rather than only seeing the effect of combinations.
This matters because naive testing of many variables together can make it impossible to tell which change actually drove a result. With an orthogonal design, statisticians can use fractional factorial methods to test many combinations with fewer total variants than a full factorial test would require, while still being able to back out the contribution of each factor.
For example, a team testing three elements, headline, button color, and hero image, each with two versions, would need eight variants for a full factorial test. An orthogonal design might reduce that to four carefully chosen combinations that still let analysts estimate the main effect of each factor, at the cost of losing some ability to detect interaction effects between them.
The main caveat is that orthogonal designs trade statistical efficiency for resolution: because not every combination is tested, interaction effects can be partially or fully confounded with main effects. Teams typically choose orthogonal designs when they want quick signal on several independent factors and have reason to believe interactions are minimal.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Experiment Decay
The gradual erosion of a winning test's uplift over time after it is fully rolled out.
See this in the wild
ABWatcher watches how top teams apply orthogonal test design.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.