TermCraft conventionFoundational
Statistical Significance
The difference is probably real, not luck.
Definition
A result unlikely to have arisen by chance under the null hypothesis, at a stated threshold.
Craft convention. True because the industry converged on it. Breaking it costs familiarity, not correctness.
In plain language
The difference is probably real, not luck.
Example
A 0.3 percent lift can be statistically significant with enough traffic and still not be worth shipping.
Continue from here
Each link says what the connection is, so you can tell a principle from an alternative from a thing people mix this up with.
Deeper reading
The full entry behind this definition.
- Confirmation BiasCognitive bias
- A/B TestingResearch method