All terms

CRO Glossary

Orthogonal Test Design

A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

Orthogonal test design is an approach to structuring multivariate experiments so that each factor being tested varies independently of the others across the full set of combinations shown to users. The goal is to be able to isolate the effect of each individual element rather than only seeing the effect of combinations.

This matters because naive testing of many variables together can make it impossible to tell which change actually drove a result. With an orthogonal design, statisticians can use fractional factorial methods to test many combinations with fewer total variants than a full factorial test would require, while still being able to back out the contribution of each factor.

For example, a team testing three elements, headline, button color, and hero image, each with two versions, would need eight variants for a full factorial test. An orthogonal design might reduce that to four carefully chosen combinations that still let analysts estimate the main effect of each factor, at the cost of losing some ability to detect interaction effects between them.

The main caveat is that orthogonal designs trade statistical efficiency for resolution: because not every combination is tested, interaction effects can be partially or fully confounded with main effects. Teams typically choose orthogonal designs when they want quick signal on several independent factors and have reason to believe interactions are minimal.

Related terms

See this in the wild

ABWatcher watches how top teams apply orthogonal test design.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.