TRUE: 一种用于信息检索中基于大语言模型的相关性判断的可重复框架
TRUE: A Reproducible Framework for LLM-Driven Relevance Judgment in Information Retrieval
AI总结：
TRUE是一种用于信息检索中基于大语言模型的相关性判断的可重复框架，通过迭代数据采样和推理评估多个相关性因素，提升相关性判断的可靠性和可重复性。
摘要：
基于大语言模型的相关性判断生成已成为推动信息检索（IR）评估方法发展的重要方法。其发展显著，经常显示出与人类判断高度相关，如LLMJudge排行榜所反映的那样\cite{rahmani2025judging}。然而，现有的相关性判断方法依赖于敏感的提示策略，缺乏标准化的工作流程来生成可靠的标签。为了填补这一空白，我们重新引入我们的方法，即任务感知的评分表评估（TRUE），用于相关性判断生成。最初为搜索会话中的有用性评估而开发，我们扩展TRUE以弥补由于其已证明的有效性和可重复的工作流程而存在的相关性判断差距。该框架利用迭代数据采样和推理来评估相关性判断，涵盖多个因素，包括意图、覆盖范围、特异性、准确性和有用性。在本文中，我们评估TRUE在TREC DL 2019、2020和LLMJudge数据集上的表现，并我们的结果表明TRUE在系统排名LLM排行榜上实现了强劲的性能。本工作的主要重点是提供一个可重复的基于大语言模型的相关性判断框架，我们进一步分析TRUE在多个维度上的有效性。
英文摘要：
LLM-based relevance judgment generation has become a crucial approach in advancing evaluation methodologies in Information Retrieval (IR). It has progressed significantly, often showing high correlation with human judgments as reflected in LLMJudge leaderboards \cite{rahmani2025judging}. However, existing methods for relevance judgments, rely heavily on sensitive prompting strategies, lacking standardized workflows for generating reliable labels. To fill this gap, we reintroduce our method, \textit{Task-aware Rubric-based Evaluation} (TRUE), for relevance judgment generation. Originally developed for usefulness evaluation in search sessions, we extend TRUE to mitigate the gap in relevance judgment due to its demonstrated effectiveness and reproducible workflow. This framework leverages iterative data sampling and reasoning to evaluate relevance judgments across multiple factors including intent, coverage, specificity, accuracy and usefulness. In this paper, we evaluate TRUE on the TREC DL 2019, 2020 and LLMJudge datasets and our results show that TRUE achieves strong performance on the system-ranking LLM leaderboards. The primary focus of this work is to provide a reproducible framework for LLM-based relevance judgments, and we further analyze the effectiveness of TRUE across multiple dimensions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
