JudgeBlender：融合判断用于自动相关性评估
JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
AI总结：
JudgeBlender通过融合多个开源LLM或多个提示的评估结果，提供自动相关性判断，在LLMJudge基准上达到与大型模型相当的竞争力，表明大型模型并非可靠评估所必需。
摘要：
检索系统的有效训练和评估需要大量的相关性判断，这些判断传统上由人工评估员收集——这一过程既昂贵又耗时。大型语言模型（LLMs）在生成搜索任务的相关性标签方面显示出潜力，为人工评估提供了一种潜在的替代方案。当前的方法通常依赖单一LLM，如GPT-4，尽管有效，但成本高昂且容易产生模型内偏差，这种偏差可能偏向于利用类似模型的系统。在本工作中，我们引入了JudgeBlender，一个采用较小规模的开源模型，通过结合多个LLM（LLMBlender）或多个提示（PromptBlender）的评估来提供相关性判断的框架。通过利用LLMJudge基准[18]，我们将JudgeBlender与最先进的方法以及LLMJudge挑战赛中的顶尖表现者进行比较。我们的结果表明，JudgeBlender取得了具有竞争力的性能，证明了对于可靠的相关性评估，非常大的模型往往并非必要。
英文摘要：
The effective training and evaluation of retrieval systems require a substantial amount of relevance judgments, which are traditionally collected from human assessors -- a process that is both costly and time-consuming. Large Language Models (LLMs) have shown promise in generating relevance labels for search tasks, offering a potential alternative to manual assessments. Current approaches often rely on a single LLM, such as GPT-4, which, despite being effective, are expensive and prone to intra-model biases that can favour systems leveraging similar models. In this work, we introduce JudgeBlender, a framework that employs smaller, open-source models to provide relevance judgments by combining evaluations across multiple LLMs (LLMBlender) or multiple prompts (PromptBlender). By leveraging the LLMJudge benchmark [18], we compare JudgeBlender with state-of-the-art methods and the top performers in the LLMJudge challenge. Our results show that JudgeBlender achieves competitive performance, demonstrating that very large models are often unnecessary for reliable relevance assessments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
