Product Analytics
Measuring what people do at scale, which tells you where problems are and never why they exist.
Definition
Product analytics records user actions as events and aggregates them. It is comprehensive about behaviour within the product and silent about intention, context and everything that happens outside it.
Craft convention. True because the industry converged on it. Breaking it costs familiarity, not correctness.
On this plate
Reach for this when
- Identifying where to focus research.
- Measuring the effect of a change.
- Understanding usage distribution across features.
- You need to know how widespread a problem is before spending research time on it.
Common mistakes
- Concluding why from behavioural data.
- Aggregate figures across distinct segments.
- Instrumenting without a taxonomy.
- Ignoring people who never entered the funnel.
- Reading a metric that moved without checking what else shipped that week.
What analytics can and cannot answer
- Can: how many, how often, in what order, where people stop.
- Can: whether a change moved a number.
- Cannot: why people stopped.
- Cannot: what people wanted and did not find.
- Cannot: who never arrived at all.
Practical rules
- Design the event taxonomy before instrumenting. Renaming events after the fact loses history.
- Segment before concluding. An aggregate median usually hides two distinct populations.
- Watch for survivorship bias: everyone in your data is someone who got that far.
- Pair every surprising number with qualitative work before acting on it.
- Respect consent and data minimisation obligations. Collect what you will use.
Making the data trustworthy
- 1
Name events for what the user did
Checkout completed, not button_click_47. The name is the documentation, and it outlives whoever wrote it.
- 2
Write the tracking plan before the code
One table of events, properties and their meaning, owned by someone. Instrumentation added ad hoc becomes unreadable within a quarter.
- 3
Segment before concluding
An average across new and returning users often describes nobody. Split first, then look.
- 4
Check what the blockers remove
Ad blockers, privacy settings and consent refusals all take a bite out of the sample, and not a random one.
- 5
Pair every number with a question
Analytics localises a problem. Sessions, interviews or tests are what explain it.
In practice
- i.
The feature nobody used, and why
Where analytics stopsUsage of a new export feature sat near zero. The data could show that people reached the screen and left, and it took four sessions to learn that the file it produced could not be opened by the tool everyone used next.
Commonly confused with
Close neighbours that get substituted for each other. The difference is stated first, then the condition that picks one over the other.
Funnel Analysis
Research methodCompletion measured step by step through one defined sequence.
Key difference
Analytics is the whole record of behaviour in the product. A funnel is one view of it: completion across a defined sequence. The funnel is the sharper tool and the blinder one, because it cannot see anyone who never entered it.
Use Product Analytics when
- You do not yet know which sequence is the problem.
- The question spans behaviour, retention or feature use.
- You need to find where to look before measuring a path.
Use Funnel Analysis when
- The steps are known, ordered and worth improving.
- The question is where people drop out of a specific flow.
- You will also account for the people who never started.
Local chart
RES·010 · Region IV · Grid G2
This entry and the entries it is stated to connect with. Each line is labelled with what the connection means.
Sources
Where a source establishes something narrower than the popular reading of it, the note says so.
- [1]
Lean Analytics
A. Croll and B. Yoskovitz, O'Reilly, 2013
On choosing the one metric that matters at a given stage, and on why most dashboards measure comfort rather than progress.
Continue from here
Each link says what the connection is, so you can tell a principle from an alternative from a thing people mix this up with.
Applies when you are designing
Principles behind this
The reasoning this solution is an application of.
- Availability HeuristicBIAS·011 · Cognitive bias
- Survivorship BiasBIAS·010 · Cognitive bias
- Help and DocumentationHEUR·010 · Heuristic
- Pareto PrinciplePRIN·002 · Principle
Often used with
These usually appear in the same screen or the same decision.
Short definition
The one-paragraph version, for when that is all you need.
- ActivationTERM·103 · Term
- BenchmarkTERM·031 · Term
- CohortTERM·101 · Term
- North Star MetricTERM·032 · Term
- Quantitative ResearchTERM·020 · Term
- RetentionTERM·104 · Term
- TriangulationTERM·023 · Term
- Vanity MetricTERM·033 · Term
Related concept
Connected closely enough to change how you apply this.
- A/B TestingRES·009 · Research method
- SurveysRES·004 · Research method
- User InterviewsRES·003 · Research method
- Measuring UXFLOW·011 · Workflow
- Prioritising Design WorkFLOW·006 · Workflow