ChatEval：通过多智能体辩论打造更优的基于 LLM 的评估器
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
AI总结：
本文提出 ChatEval 多智能体辩论框架，让多个 LLM 裁判协作讨论并评估生成文本，以更接近人类评估质量。
摘要：
文本评估历来面临重大挑战，往往需要大量人力与时间成本。随着大型语言模型（LLM）的出现，研究人员探索了将 LLM 作为人工评估替代方案的潜力。尽管这些基于单智能体的方法展现出前景，但实验结果表明，仍需进一步改进，以缩小其当前有效性与人类水平评估质量之间的差距。我们认识到，人工评估流程的最佳实践通常涉及多名人工标注者协作完成评估，因此借助多智能体辩论框架，超越单智能体提示策略。这种基于多智能体的方法使一组 LLM 能够与众多智能同伴协同，利用它们各自不同的能力和专业知识，提高处理复杂任务的效率与效果。本文构建了一个名为 ChatEval 的多智能体裁判团队，用于针对开放式问题和传统自然语言生成（NLG）任务，自主讨论并评估不同模型所生成回复的质量。分析表明，ChatEval 超越了单纯的文本打分，提供了一种模仿人类的评估流程以实现可靠评估。代码位于 https://github.com/chanchimin/ChatEval。
英文摘要：
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at https://github.com/chanchimin/ChatEval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
