Банк вопросов из реальных собеседований

100 independent tests were conducted with a significance level of 5%, and two gave a p-value below 0.05. What can we conclude?

Minimizing squared error corresponds to maximum likelihood under what noise distribution, and why?

Describe how you would train and validate a transformer-style reranking model for marketplace recommendations.

How does mixed precision training work in PyTorch, what is the difference between FP16 and BF16, and what else is commonly needed beyond autocast?

Which online metrics should I choose for the A/B test of the card with the generated description and which guardrails should I install?

There are N0 negative and N1 positive objects, and the classifier gives everyone one probability p. Which p minimizes the binary logarithmic loss function?

In the A/B test, the conversion of control and experimental groups is compared. What is the minimum sample size required to detect a statistically significant effect?

How to build reporting around a LLM agent to understand quality, benefits, errors and costs?

What offline metrics and manual assessment can be used to check the quality of generated object responses before the A/B test?

How to build a reproducible system of offline evaluation of the new model of recommendations and link its results with the subsequent A/B test?

How would you evaluate candidate generation and reranking offline for an item-to-item fashion recommender before an A/B test?

What does the Central Limit Theorem say and why is it important in statistics and A/B testing?

In PyTorch DDP training, which common layer can behave badly across processes and how do teams usually handle it?

A new perception detector improves some offline metrics but degrades others. How do you decide whether to ship it to production?

What properties should embeddings have to search, recommend, or match objects?

How to take into account seasonality in search and how to launch a new model in an online experiment?

Explain why statistical significance is needed in A/B tests, what a p-value means, and what affects whether an experiment is significant.

How to choose negative examples when training a recommendation model and why use objects from the same batch?

A disease has prevalence 1%. A test is 99% accurate for both sick and healthy people. If the test is positive, what is the probability that the person is actually sick?

The disease occurs in 1% of people, and the diagnostic test is mistaken in 5% of cases. If the result is positive, how do you calculate the probability that a person is really sick, and where are they most often mistaken?

A binary image classifier is trained with BCE loss. On validation, accuracy rises but BCE loss also rises. Can this happen and what are plausible causes?

There are 1000 coins, one of them has an eagle on both sides, the rest are fair. We chose a coin at random and got 10 heads in a row. What is the probability that a counterfeit coin is chosen?

How are the central limit theorem, A/B test design, and MDE related?

How to explain the p-value without calling it the probability of the null hypothesis being true?