基于LLM的相关性判断方法基准测试
Benchmarking LLM-based Relevance Judgment Methods
AI总结：
本文针对基于LLM的相关性评估研究缺乏替代方法探索和全面比较的问题，系统比较了二元、分级、成对偏好及两种信息块评估法，在多数据集验证并发布模型判断，为该领域提供全面基准。
摘要：
大型语言模型（LLMs）正越来越多地应用于学术和工业场景，以自动化信息检索系统的评估，尤其是通过生成分级相关性判断。此前关于基于LLM的相关性评估的研究主要聚焦于通过各种提示策略复现人工分级相关性判断。然而，对替代评估方法或全面比较研究的探索有限。本文系统比较了多种基于LLM的相关性评估方法，包括二元相关性判断、分级相关性评估、基于成对偏好的方法，以及两种基于信息块的评估方法——文档无关型和文档依赖型。除了使用肯德尔相关系数基于系统排名进行传统比较外，我们还考察了LLM判断与从相关性等级推断出的人类偏好的对齐程度。我们在来自TREC深度学习赛道2019、2020、2021年的数据集以及专注于非事实型开放域问答的ANTIQUE数据集上进行了广泛实验。作为数据发布的一部分，我们包含了开源模型（Llama3.2b）和商业模型（gpt-4o）生成的相关性判断。我们的目标是复现各种基于LLM的相关性判断方法，以提供全面比较。所有代码、数据和资源均可在我们的GitHub仓库获取：https://github.com/Narabzad/llm-relevance-judgement-comparison。
英文摘要：
Large Language Models (LLMs) are increasingly deployed in both academic and industry settings to automate the evaluation of information seeking systems, particularly by generating graded relevance judgments. Previous work on LLM-based relevance assessment has primarily focused on replicating graded human relevance judgments through various prompting strategies. However, there has been limited exploration of alternative assessment methods or comprehensive comparative studies. In this paper, we systematically compare multiple LLM-based relevance assessment methods, including binary relevance judgments, graded relevance assessments, pairwise preference-based methods, and two nugget-based evaluation methods~--~document-agnostic and document-dependent. In addition to a traditional comparison based on system rankings using Kendall correlations, we also examine how well LLM judgments align with human preferences, as inferred from relevance grades. We conduct extensive experiments on datasets from three TREC Deep Learning tracks 2019, 2020 and 2021 as well as the ANTIQUE dataset, which focuses on non-factoid open-domain question answering. As part of our data release, we include relevance judgments generated by both an open-source (Llama3.2b) and a commercial (gpt-4o) model. Our goal is to \textit{reproduce} various LLM-based relevance judgment methods to provide a comprehensive comparison. All code, data, and resources are publicly available in our GitHub Repository at https://github.com/Narabzad/llm-relevance-judgement-comparison.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
