大语言模型用于评估是否可信？通过智能体辩论实现作为评估器的LLM的可扩展元评估
Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
AI总结：
针对LLM作为评估器的元评估受基准限制、需大量人工标注的问题，提出基于智能体辩论的ScaleEval框架，通过多LLM智能体多轮讨论辅助人工筛选优质评估器，降低标注工作量并实现可扩展元评估。
摘要：
尽管大语言模型（LLM）在广泛的任务和场景中具备实用性，但开发一种能在多样语境下可靠评估LLM的方法仍具挑战性。现代评估方法常使用LLM来评判LLM生成的回答。然而，为评估这些作为评估器的LLM的有效性而开展的元评估，通常受限于现有基准的覆盖范围，或需要大量人工标注。这凸显了可扩展元评估方法的紧迫性，这类方法需能有效、可靠且高效地评估LLM作为评估器在各类任务和场景下的表现，尤其是在潜在的全新用户自定义场景中。为填补这一空白，我们提出ScaleEval——一种智能体辩论辅助的元评估框架，它利用多个可交互的LLM智能体的能力。该框架支持多轮讨论，协助人工标注者识别能力最强的LLM评估器，大幅减轻了以往元评估中需要大规模标注时的工作负担。我们已发布该框架的代码，公开地址为：https://github.com/GAIR-NLP/scaleeval。
英文摘要：
Despite the utility of Large Language Models (LLMs) across a wide range of tasks and scenarios, developing a method for reliably evaluating LLMs across varied contexts continues to be challenging. Modern evaluation approaches often use LLMs to assess responses generated by LLMs. However, the meta-evaluation conducted to assess the effectiveness of these LLMs as evaluators is typically constrained by the coverage of existing benchmarks or requires extensive human annotation. This underscores the urgency of methods for scalable meta-evaluation that can effectively, reliably, and efficiently evaluate the performance of LLMs as evaluators across diverse tasks and scenarios, particularly in potentially new, user-defined scenarios. To fill this gap, we propose ScaleEval, an agent-debate-assisted meta-evaluation framework that leverages the capabilities of multiple communicative LLM agents. This framework supports multi-round discussions to assist human annotators in discerning the most capable LLMs as evaluators, which significantly eases their workload in cases that used to require large-scale annotations during meta-evaluation. We release the code for our framework, which is publicly available at: \url{https://github.com/GAIR-NLP/scaleeval}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
