基于标准的大语言模型相关性判定
Criteria-Based LLM Relevance Judgments
AI总结：
提出Multi-Criteria框架，将相关性分解为精确性、覆盖度、主题性和上下文契合度等标准，提升LLM相关性判定的鲁棒性与可解释性，并在TREC数据集上验证了其优于直接评分方法。
摘要：
相关性判定对于评估信息检索系统至关重要，但传统的人工标注标签既耗时又昂贵。因此，许多研究人员转向自动替代方案以加速方法开发。其中，大语言模型（LLMs）通过提示直接生成相关性标签，提供了一种可扩展的解决方案。然而，在没有约束的情况下提示LLM给出相关性标签，往往不仅会导致错误的预测，还会产生人类难以解释的输出。我们提出了基于LLM相关性判定的Multi-Criteria框架，将相关性的概念分解为多个标准——例如精确性、覆盖度、主题性和上下文契合度——以提升检索评估的鲁棒性和可解释性。我们在三个数据集上验证了该方法：2019年和2020年的TREC Deep Learning tracks，以及LLMJudge（基于TREC DL 2023）。结果表明，Multi-Criteria判定增强了系统排名/排行榜性能。此外，我们强调了该方法相对于直接评分方法的优势与局限，为未来信息检索自动评估框架的开发提供了指导。
英文摘要：
Relevance judgments are crucial for evaluating information retrieval systems, but traditional human-annotated labels are time-consuming and expensive. As a result, many researchers turn to automatic alternatives to accelerate method development. Among these, Large Language Models (LLMs) provide a scalable solution by generating relevance labels directly through prompting. However, prompting an LLM for a relevance label without constraints often results in not only incorrect predictions but also outputs that are difficult for humans to interpret. We propose the Multi-Criteria framework for LLM-based relevance judgments, decomposing the notion of relevance into multiple criteria--such as exactness, coverage, topicality, and contextual fit--to improve the robustness and interpretability of retrieval evaluations compared to direct grading methods. We validate this approach on three datasets: the TREC Deep Learning tracks from 2019 and 2020, as well as LLMJudge (based on TREC DL 2023). Our results demonstrate that Multi-Criteria judgments enhance the system ranking/leaderboard performance. Moreover, we highlight the strengths and limitations of this approach relative to direct grading approaches, offering insights that can guide the development of future automatic evaluation frameworks in information retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
