Вопрос по метрикам
How do you design an A/B test, estimate sample size/MDE, and handle cases where treatment and control users are not independent, such as drivers and passengers in a marketplace?
Ответить самому
Сначала сформулируйте ответ как на собеседовании, затем откройте разбор и оцените себя.
Короткий ответ
Define hypothesis, unit, metric, alpha, power and MDE. If units interfere, avoid naive user randomization and consider cluster, geo or time switchback designs.
Полный разбор
A classical A/B test defines the hypothesis, treatment, randomization unit, primary metric, guardrails, alpha, power, MDE, duration and stopping rule. Sample size depends on baseline rate or variance, MDE, significance level, power and traffic.
In marketplaces, user-level randomization can violate independence. If one group of drivers receives a change, passengers and untreated drivers may still be affected through supply, demand, prices, waiting times or allocation.
Alternatives include cluster randomization by geography or marketplace cell, geo experiments, time-based switchback tests where the whole market alternates between variants, saturation designs and diff-in-diff/synthetic control when clean randomization is impossible. CUPED reduces variance, but does not remove interference.
Теория
The experiment unit must match the unit where interference is acceptably low.
Типичные ошибки
- Use individual randomization when users compete for the same supply.
- Claim CUPED fixes network effects.
- Ignore guardrails like wait time and cancellations.
Как отвечать на собеседовании
- Say "interference" or "network effects" explicitly.
- Offer switchback or geo randomization as concrete alternatives.