One-Tailed vs Two-Tailed Test

A two-tailed test checks whether a variant is different from control in either direction — better or worse — and is the standard, more conservative default in most A/B testing tools. A one-tailed test only checks whether the variant is better (or only worse) than control, which requires less evidence to declare significance for the same confidence level, but it means you've explicitly given up the ability to detect or report a result in the opposite direction.

The choice matters because switching to a one-tailed test partway through analysis, just because the two-tailed result wasn't quite significant, is a form of p-hacking. The decision needs to be made and documented before the test starts, based on a genuine belief that a decrease is either impossible or irrelevant to the decision at hand.

For example, a team testing whether adding a trust badge to a checkout page increases conversion might argue a one-tailed test is appropriate because they have no interest in acting on a result showing the badge hurts conversion — they'd simply not ship it either way. But if there's a real business reason to know whether a change might backfire, a two-tailed test is the safer, more standard choice.