DEBATE：基于魔鬼代言人的评估与文本评价
DEBATE: Devil's Advocate-Based Assessment and Text Evaluation
AI总结：
针对NLG评估中单一LLM智能体存在偏差的问题，提出基于魔鬼代言人的多智能体辩论框架DEBATE，在SummEval和TopicalChat基准上显著超越现有方法。
摘要：
随着自然语言生成（NLG）模型的普及，系统性地评估机器生成文本的质量变得日益重要。近期研究引入了基于大语言模型（LLM）的评估器，这些评估器作为无参考指标运行，展现出其熟练处理新任务的能力。然而，这些模型通常依赖单一智能体方法，我们认为这给其性能带来了固有局限。这是因为LLM智能体的响应中存在偏差，包括对特定文本结构或内容的偏好。在本工作中，我们提出了DEBATE，一个基于多智能体评分系统并融入魔鬼代言人概念的NLG评估框架。在该框架内，一个智能体被指示去批评其他智能体的论点，从而可能解决LLM智能体回答中的偏差。DEBATE在NLG评估的两个元评估基准SummEval和TopicalChat上大幅超越了先前的最先进方法。我们还表明，智能体之间辩论的广泛程度以及智能体的人格设定能够影响评估器的性能。
英文摘要：
As natural language generation (NLG) models have become prevalent, systematically assessing the quality of machine-generated texts has become increasingly important. Recent studies introduce LLM-based evaluators that operate as reference-free metrics, demonstrating their capability to adeptly handle novel tasks. However, these models generally rely on a single-agent approach, which, we argue, introduces an inherent limit to their performance. This is because there exist biases in LLM agent's responses, including preferences for certain text structure or content. In this work, we propose DEBATE, an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil's Advocate. Within the framework, one agent is instructed to criticize other agents' arguments, potentially resolving the bias in LLM agent's answers. DEBATE substantially outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. We also show that the extensiveness of debates among agents and the persona of an agent can influence the performance of evaluators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
