Вопрос по метрикам
Explain why statistical significance is needed in A/B tests, what a p-value means, and what affects whether an experiment is significant.
Ответить самому
Сначала сформулируйте ответ как на собеседовании, затем откройте разбор и оцените себя.
Короткий ответ
Significance helps distinguish real effect from random noise. A p-value is the probability of seeing data at least this extreme under the null hypothesis, and it depends on effect size, variance, sample size and test design.
Полный разбор
In an A/B test, observed metric differences can appear by chance. Statistical significance gives a disciplined way to decide whether the observed difference is too unlikely under the null hypothesis of no effect.
A p-value is not the probability that the feature works. It is the probability, assuming the null hypothesis is true, of observing a result at least as extreme as the one measured. If p-value is below a prechosen alpha such as 0.05, we reject the null at that significance level.
Whether an experiment becomes significant depends on effect size, sample size, metric variance, traffic allocation, duration, test choice, multiple testing, guardrails and data quality. More samples generally reduce uncertainty, but biased traffic or broken randomization cannot be fixed just by waiting longer.
Теория
A/B testing is about uncertainty around causal effect estimates, not just comparing two observed averages.
Типичные ошибки
- Define p-value as probability the hypothesis is true.
- Use 5% as a magic truth boundary.
- Ignore power and sample size planning.
- Forget variance, randomization and multiple testing.
Как отвечать на собеседовании
- Say “under the null hypothesis” when defining p-value.
- Name effect size and sample size as key drivers.