Skip to content
UX Atlas
Research methodEstablished researchAdvanced

A/B Testing

Comparing variants with real traffic, which tells you what happened without telling you why.

Definition

An A/B test randomly assigns users to variants and compares a predefined metric. Done properly it gives causal evidence about the effect of a change on that metric, within the tested population and period.

Established research. Originates in published research and has been replicated. The effect is real. Its size in your product still needs measuring.

On this page
  1. Definition
  2. Requirements for a valid test
  3. In practice
  4. Sources
  5. Continue from here

Requirements for a valid test

  1. 1.Calculate the sample size in advance from the baseline rate and the smallest effect worth detecting.
  2. 2.Fix the stopping rule before starting. Stopping when the result looks good inflates false positives substantially.
  3. 3.Run for whole weeks, so weekday and weekend behaviour are both represented.
  4. 4.Define one primary metric. Testing many metrics without correction guarantees spurious significance.
  5. 5.Watch guardrail metrics for damage the primary metric would not show.
  6. 6.Check for a novelty effect by comparing early and late periods within the test.

In practice

Peeking

Experiment hygiene

Checking results daily and stopping at first significance can raise the false positive rate several times above the nominal level. Either fix the sample size in advance or use a sequential method designed for continuous monitoring.

Sources

Where a source establishes something narrower than the popular reading of it, the note says so.

  • Trustworthy Online Controlled Experiments

    R. Kohavi, D. Tang and Y. Xu, Cambridge University Press, 2020

Continue from here

Each link says what the connection is, so you can tell a principle from an alternative from a thing people mix this up with.

Applies when you are designing

A CheckoutA Pricing PageAn Ecommerce Experience

Principles behind this

The reasoning this solution is an application of.

Often used with

These usually appear in the same screen or the same decision.

Commonly confused with

Close enough to be mixed up, different enough to matter.

  • Usability TestingResearch method

    One tells you why something fails with five people. The other tells you which version wins with thousands, and not why.

Short definition

The one-paragraph version, for when that is all you need.

Related concept

Connected closely enough to change how you apply this.