Вопрос по метрикам
How would you evaluate and improve a summarization service if user feedback is sparse or unavailable?
Ответить самому
Сначала сформулируйте ответ как на собеседовании, затем откройте разбор и оцените себя.
Короткий ответ
Use a layered eval: reference-based metrics where labels exist, human or assessor rubrics, LLM-as-judge with source-grounding checks, slice metrics and production proxies such as edits, copies and retention.
Полный разбор
If reference summaries exist, ROUGE/BERTScore-like metrics can provide a quick signal, but they do not fully capture faithfulness or usefulness. For open-ended summarization, create a human rubric: factuality, coverage of main points, concision, readability, harmful omissions and hallucinations.
LLM-as-judge can scale evaluation if used carefully. Give the judge the source and summary, ask it to score factual consistency, missing key points and verbosity, and calibrate it against human judgments. Use a stronger or different model than the production summarizer where possible.
Segment evaluation by URL versus raw text, language, content length, domain, extraction confidence and user intent. If explicit feedback is sparse, collect implicit signals such as copy/share, edits after summary, regeneration, dwell, thumbs-up prompts on samples and support complaints.
Теория
Summarization has no single sufficient metric; robust evaluation combines references, humans, judges, slices and product proxies.
Типичные ошибки
- Use ROUGE as the only success metric.
- Let an LLM judge without source text or calibration.
- Ignore hallucination and factual consistency.
- Aggregate across short text, long pages and noisy web pages.
Как отвечать на собеседовании
- Use “faithfulness, coverage, concision” as rubric anchors.
- Mention calibration of LLM-as-judge against humans.