CRO Glossary
Qualitative Usability Testing
Observing a small number of real users complete tasks on a product to surface friction and comprehension issues before or alongside quantitative A/B tests.
Qualitative usability testing involves watching a handful of representative users (often 5-8 per round) attempt real tasks on a site or prototype while thinking aloud, so researchers can see exactly where they hesitate, misread copy, or abandon a flow. Unlike a running A/B test, it doesn't require significant traffic or a live split; it's typically done moderated (a facilitator asks questions live) or unmoderated (users record themselves completing tasks asynchronously).
CRO teams use it to generate and prioritize hypotheses before committing traffic to a formal experiment. Usability sessions reveal why users struggle, which pure conversion metrics can't explain on their own — a drop in checkout completion might look identical in the data whether the cause is a confusing error message or a broken payment field, but five minutes of watching someone try to check out makes the cause obvious.
A concrete example: before testing a new checkout flow, a team runs five moderated sessions and discovers that three of five participants don't notice the discount code field because it's visually deprioritized. That insight directly shapes the design of the A/B test variant, rather than the team guessing at what to change.
Because sample sizes are small, usability testing findings aren't statistically generalizable the way a properly powered experiment is — it's a complementary, hypothesis-generating tool rather than a replacement for quantitative validation, and is often paired with qualitative research and friction audits earlier in the experiment design process.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply qualitative usability testing.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.