Measuring UX
Pick one behavioural measure per goal, pair it with a quality measure, and decide the threshold before you look.
Definition
Measuring experience means choosing indicators that move when the experience changes, and interpreting them without pretending correlation is proof. The common failure is measuring what is easy to collect rather than what would change a decision.
Mixed evidence. Real research exists, but findings conflict, effect sizes are small, or the jump from the lab to interface design is contested.
On this plate
Reach for this when
- Committing to a redesign and needing a before and after.
- Choosing what to report to stakeholders regularly.
- An experiment result is about to be over-read.
- A redesign is about to start and there is no before to compare the after against.
Common mistakes
- Measuring what is easy to collect.
- No baseline.
- Thresholds decided after seeing the data.
- A single metric with no counterweight.
- Starting to measure after the change has shipped, which leaves the improvement unprovable.
Measures and what they actually tell you
| Measure | Tells you | Misleads when |
|---|---|---|
| Task success rate | Whether people can complete the task at all | The task was defined generously, or helped along by a moderator. |
| Time on task | Efficiency for known tasks | Longer is better, as in reading or browsing. |
| Error rate | Where the interface misleads | Errors are recovered so smoothly nobody notices them. |
| SUS or UMUX-Lite | A comparable satisfaction score with published benchmarks | Sample is small, or the questionnaire is run immediately after help. |
| Adoption and retention | Whether the change stuck | A launch spike is read as sustained behaviour. |
| Support contact rate | Where the product fails to explain itself | Support channels changed at the same time. |
Setting up a measure you can trust
- 1
Start from the decision
What would you do differently at a good result and at a bad one? If the answer is the same, do not collect it.
- 2
Pair behaviour with quality
Completion without satisfaction hides a grim experience people endured. Satisfaction without completion hides a pleasant failure.
- 3
Set the threshold in advance
Decide what counts as success before the data arrives, because afterwards every number can be argued into a win.
- 4
Take a baseline
Measure before shipping. A post-launch number with nothing to compare it to settles no argument.
- 5
Watch the counterweight
Every metric can be moved by damaging something else. Name the measure that would catch that and watch it too.
Reporting a measure
Do
- Give the sample size, the date range and how people were selected, every time.
- Show the distribution, not only the average. Two clusters do not have a middle.
- State the measurement error, so a two point move is not read as progress.
- Pair each measure with a guardrail it must not damage.
Do not
- Compare a number to one gathered under different conditions.
- Report a satisfaction score with no task behind it.
- Let a metric become a target without deciding how it will be gamed.
- Track something nobody has agreed to act on.
In practice
- i.
A metric moved by damage
Session length rose after a navigation change, which was reported as engagement. Task completion had fallen: people were taking longer because they could not find things. The counterweight measure caught it, the headline measure did not.
- ii.
The average that described nobody
DistributionsA mean task time of ninety seconds came from a group who finished in thirty and a group who gave up at three minutes. The average was the one number in the report that matched no participant.
Local chart
FLOW·011 · Region IV · Grid G1
This entry and the entries it is stated to connect with. Each line is labelled with what the connection means.
Sources
Where a source establishes something narrower than the popular reading of it, the note says so.
- [1]
Measuring the User Experience
Tullis and Albert, 2013
Standard reference for task-based measures and their interpretation.
- [2]
SUS: A quick and dirty usability scale
J. Brooke, 1996
Origin of the most widely benchmarked satisfaction questionnaire.
Continue from here
Each link says what the connection is, so you can tell a principle from an alternative from a thing people mix this up with.
Principles behind this
The reasoning this solution is an application of.
- Confirmation BiasBIAS·009 · Cognitive bias
- Survivorship BiasBIAS·010 · Cognitive bias
- Product ThinkingMOD·002 · Mental model
Related concept
Connected closely enough to change how you apply this.
- A/B TestingRES·009 · Research method
- Funnel AnalysisRES·011 · Research method
- Product AnalyticsRES·010 · Research method
- Usability TestingRES·001 · Research method
- Problem FramingFLOW·007 · Workflow