Банк вопросов из реальных собеседований

What offline and online metrics do a recommendation system need with visual and text features?

What metrics to use to evaluate rankings in recommendations or search?

How do you know if users like content on a news feed? What data to monitor and what biases can distort these metrics?

For a fraud-detection model, how would you choose validation metrics and a decision threshold when false positives and false negatives have different business costs?

100 independent tests were conducted with a significance level of 5%, and two gave a p-value below 0.05. What can we conclude?

Describe how you would train and validate a transformer-style reranking model for marketplace recommendations.

How does mixed precision training work in PyTorch, what is the difference between FP16 and BF16, and what else is commonly needed beyond autocast?

Which online metrics should I choose for the A/B test of the card with the generated description and which guardrails should I install?

There are N0 negative and N1 positive objects, and the classifier gives everyone one probability p. Which p minimizes the binary logarithmic loss function?

In the A/B test, the conversion of control and experimental groups is compared. What is the minimum sample size required to detect a statistically significant effect?

How to build reporting around a LLM agent to understand quality, benefits, errors and costs?

How would you evaluate candidate generation and reranking offline for an item-to-item fashion recommender before an A/B test?

What offline metrics and manual assessment can be used to check the quality of generated object responses before the A/B test?

How to build a reproducible system of offline evaluation of the new model of recommendations and link its results with the subsequent A/B test?

In PyTorch DDP training, which common layer can behave badly across processes and how do teams usually handle it?

A new perception detector improves some offline metrics but degrades others. How do you decide whether to ship it to production?

What properties should embeddings have to search, recommend, or match objects?

How to take into account seasonality in search and how to launch a new model in an online experiment?

Explain why statistical significance is needed in A/B tests, what a p-value means, and what affects whether an experiment is significant.

How to choose negative examples when training a recommendation model and why use objects from the same batch?

A disease has prevalence 1%. A test is 99% accurate for both sick and healthy people. If the test is positive, what is the probability that the person is actually sick?

The disease occurs in 1% of people, and the diagnostic test is mistaken in 5% of cases. If the result is positive, how do you calculate the probability that a person is really sick, and where are they most often mistaken?

A binary image classifier is trained with BCE loss. On validation, accuracy rises but BCE loss also rises. Can this happen and what are plausible causes?

There are 1000 coins, one of them has an eagle on both sides, the rest are fair. We chose a coin at random and got 10 heads in a row. What is the probability that a counterfeit coin is chosen?