Банк вопросов из реальных собеседований

What is ROC-AUC, how do you build the ROC curve, and what does the area mean probabilistically?

A monthly subscription costs 10 dollars. 100% of users pay month 1, then 50%, 40%, 30%, and after that each month is 10% lower than the previous one through month 12. Estimate annual LTV.

How to determine the accuracy and completeness of a binary classifier and how they are equal for a diagnostic test of a rare disease problem?

Before launching an A/B test, what should you define? What determines duration or sample size, and how do you interpret a p-value?

How to conduct offline and online experiments for a recommendation model? What is important in the A/B test: MDE, p-value, sampling, network effects and metrics?

How can you get a sentence embedding from BERT, how do sentence transformers differ, and why is this similar to metric learning for image pairs?

There are historical users and calculated LTV. How to use bootstrap to estimate LTV variance and get a lower bound for traffic purchasing decisions?

What is bootstrap for and why doesn’t it by itself reduce the variance of the experiment?

What to check if the ranking/model metric is unexpectedly low or the model looks overfit/underfit?

What is the difference between FP16 and BF16 and why is BF16 often more stable for training?

How to train and evaluate a model if a positive class is rare?

The model works well on average, but some classes have low F1. How to diagnose and repair?

It is necessary to predict the revenue of the user for 365 days on early signs. Why can a direct approach work poorly?

How to count MAP/NDCG for recommendations and why are these metrics not enough without a business link?

How to build a system that, based on photographs of advertisements, understands that there is another car in the report and removes erroneous matches?

How to use fresh incomplete cohorts if R365 is not yet known?

How does NDCG differ from MAP and why is it difficult to directly optimize such metrics by gradient descent?

What is the difference between NDCG/MAP and pairwise losses like BPR/WARP?

What metrics to look for when rolling out a new recommendation or search model?

How to jointly explain p-value, significance level and confidence interval?

How to briefly explain p-value, where it is used, and how to derive Bayes' formula through conditional probability and total probability?

At what stage of a recommendation system is recall more important, and at what stage is accuracy more important?

A dataset has 95 ones and 5 zeros. The model always predicts one. Compute precision, recall, and ROC-AUC, then explain why the model is bad and how to choose a threshold.

How to explain Precision/Recall and what to check if the model is to be generalized to new regions or geographical features?