Skip to content
UX Atlas
Plate RES·009 · Research methodGrid H2 · Rev. 2026-08-14

A/B Testing

Comparing variants with real traffic, which tells you what happened without telling you why.

Research methodEstablished researchAdvanced

Definition

An A/B test randomly assigns users to variants and compares a predefined metric. Done properly it gives causal evidence about the effect of a change on that metric, within the tested population and period.

Established research. Originates in published research and has been replicated. The effect is real. Its size in your product still needs measuring.

On this plate
  1. 1Definition
  2. 2Requirements for a valid test
  3. 3Before you start it
  4. 4In practice
  5. 5Commonly confused with
  6. 6Local chart
  7. 7Sources
  8. 8Continue from here

Reach for this when

  • Enough traffic to reach adequate power in a reasonable time.
  • A clear primary metric.
  • The change is reversible and measurable.
  • Two designs both have a defensible argument and enough traffic exists to settle it.

Common mistakes

  • Optional stopping.
  • Underpowered tests.
  • Multiple comparisons without correction.
  • Optimising a short-term metric at the expense of trust.
  • Stopping the moment the result crosses significance, which inflates the effect you report.

Requirements for a valid test

  1. 1.Calculate the sample size in advance from the baseline rate and the smallest effect worth detecting.
  2. 2.Fix the stopping rule before starting. Stopping when the result looks good inflates false positives substantially.
  3. 3.Run for whole weeks, so weekday and weekend behaviour are both represented.
  4. 4.Define one primary metric. Testing many metrics without correction guarantees spurious significance.
  5. 5.Watch guardrail metrics for damage the primary metric would not show.
  6. 6.Check for a novelty effect by comparing early and late periods within the test.

Before you start it

  1. 1

    Write the hypothesis and the guardrails

    What you expect to move, by how much, and which metrics must not get worse while it does.

  2. 2

    Calculate the sample size in advance

    From the baseline rate and the smallest effect worth shipping. If the traffic is not there, the test is not the method.

  3. 3

    Fix the run length before launch

    Whole weeks, so weekday and weekend behaviour are both included, and no peeking that ends the test early.

  4. 4

    Check the split is real

    Compare the groups on a metric the change cannot affect. If those differ, the assignment is broken.

  5. 5

    Decide what a flat result means

    No difference is a valid and common outcome. Agree beforehand whether it means ship, revert or stop asking.

In practice

  1. i.

    Peeking

    Experiment hygiene

    Checking results daily and stopping at first significance can raise the false positive rate several times above the nominal level. Either fix the sample size in advance or use a sequential method designed for continuous monitoring.

  2. ii.

    Winning the step and losing the funnel

    Guardrails

    A simplified form lifted completion by a fifth, and the removed field had been the one that filtered out unqualified leads. The step metric improved, the metric the business cared about did not, and only the guardrail caught it.

Commonly confused with

Close neighbours that get substituted for each other. The difference is stated first, then the condition that picks one over the other.

vs.

Usability Testing

Research method

Watching people attempt real tasks to find where the interface breaks down.

Key difference

Usability testing tells you why something fails, with a handful of people watched closely. A/B testing tells you which version wins, with thousands of people and no explanation of the result.

Use A/B Testing when

  • Two defensible options remain and the argument is stuck.
  • There is enough traffic to reach significance in reasonable time.
  • The outcome is a metric you already trust.

Use Usability Testing when

  • You need the cause behind a number you already have.
  • The design is not built or shipped yet.
  • Traffic is too thin to measure anything reliably.

Learn about Usability Testing

Local chart

RES·009 · Region IV · Grid H2

This entry and the entries it is stated to connect with. Each line is labelled with what the connection means.

Confirmation BiasBIAS·009 · PRINCIPLES BEHIND THISStatus Quo BiasBIAS·005 · PRINCIPLES BEHIND THISSurvivorship BiasBIAS·010 · PRINCIPLES BEHIND THISDeceptive PatternsPRIN·011 · PRINCIPLES BEHIND THISMere Exposure EffectPSY·009 · PRINCIPLES BEHIND THISPostHogTOOL·025 · OFTEN USED WITHUsability TestingRES·001 · COMMONLY CONFUSED WITHA/B TestTERM·100 · SHORT DEFINITIONConversion RateTERM·105 · SHORT DEFINITIONEffect SizeTERM·026 · SHORT DEFINITIONFeature FlagTERM·099 · SHORT DEFINITIONQuantitative ResearchTERM·020 · SHORT DEFINITION

Sources

Where a source establishes something narrower than the popular reading of it, the note says so.

  1. [1]

    Trustworthy Online Controlled Experiments

    R. Kohavi, D. Tang and Y. Xu, Cambridge University Press, 2020

Continue from here

Each link says what the connection is, so you can tell a principle from an alternative from a thing people mix this up with.

Applies when you are designing

A CheckoutA Pricing PageAn Ecommerce Experience

Principles behind this

The reasoning this solution is an application of.

Often used with

These usually appear in the same screen or the same decision.

Short definition

The one-paragraph version, for when that is all you need.

Related concept

Connected closely enough to change how you apply this.