LLM 可修补评估中缺失的相关性判断
LLMs Can Patch Up Missing Relevance Judgments in Evaluation
AI总结：
本文提出用 LLM 按详细指令为 IR 评估中的未判断文档自动补全细粒度相关性标签，并在 TREC DL 模拟缺失判断实验中验证其与人工判断高度一致。
摘要：
信息检索基准中未经判断的文档或判断空洞在评估时会被视为不相关，因此在衡量有效性时不会带来任何增益。然而，这些缺失的判断可能会在无意中给评估引入偏差，因为它们在某个检索模型中的普遍程度严重依赖于池化过程。因此，填补空洞对于确保可靠且准确的评估至关重要。为所有文档收集人工判断既繁琐又不切实际。本文旨在利用大语言模型（LLM）自动标注未经判断的文档。我们的目标是通过详细指令指导 LLM，为空洞分配细粒度的相关性判断。为此，我们从 TREC DL tracks 的相关性判断中随机删除相关文档，系统地模拟具有不同空洞程度的场景。实验表明，我们基于 LLM 的方法与真实相关性判断之间存在很强的相关性。基于在三个 TREC DL 数据集上开展的模拟实验，在仅保留 10% 判断的极端场景下，我们的方法在 Vicuña-7B 和 GPT-3.5 Turbo 上分别取得平均 0.87 和 0.92 的 Kendall tau 相关系数。
英文摘要：
Unjudged documents or holes in information retrieval benchmarks are considered non-relevant in evaluation, yielding no gains in measuring effectiveness. However, these missing judgments may inadvertently introduce biases into the evaluation as their prevalence for a retrieval model is heavily contingent on the pooling process. Thus, filling holes becomes crucial in ensuring reliable and accurate evaluation. Collecting human judgment for all documents is cumbersome and impractical. In this paper, we aim at leveraging large language models (LLMs) to automatically label unjudged documents. Our goal is to instruct an LLM using detailed instructions to assign fine-grained relevance judgments to holes. To this end, we systematically simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our experiments reveal a strong correlation between our LLM-based method and ground-truth relevance judgments. Based on our simulation experiments conducted on three TREC DL datasets, in the extreme scenario of retaining only 10% of judgments, our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicuña-7B and GPT-3.5 Turbo respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
