CollabEval: 通过多智能体协作增强LLM作为裁判
CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration
AI总结：
CollabEval通过多智能体协作提升LLM作为裁判的评估性能，克服单一模型的不一致性和偏见问题。
摘要：
大型语言模型（LLMs）已彻底改变了AI生成内容的评估，LLM作为裁判范式日益流行。然而，当前单一LLM评估方法面临显著挑战，包括判断不一致性和预训练数据固有的偏见。为了解决这些限制，我们提出了CollabEval，一种新的多智能体评估框架，实现了三个阶段的协作评估过程：初始评估、多轮讨论和最终判断。与现有依赖竞争辩论或单模型评估的方法不同，CollabEval强调多个智能体之间的协作，并通过战略共识检查提高效率。我们的广泛实验表明，CollabEval在多个维度上一致优于单一LLM方法，即使当个体模型表现不佳时，其鲁棒性也得以保持。该框架为各种评估标准提供全面支持，同时通过其协作设计确保效率。
英文摘要：
Large Language Models (LLMs) have revolutionized AI-generated content evaluation, with the LLM-as-a-Judge paradigm becoming increasingly popular. However, current single-LLM evaluation approaches face significant challenges, including inconsistent judgments and inherent biases from pre-training data. To address these limitations, we propose CollabEval, a novel multi-agent evaluation framework that implements a three-phase Collaborative Evaluation process: initial evaluation, multi-round discussion, and final judgment. Unlike existing approaches that rely on competitive debate or single-model evaluation, CollabEval emphasizes collaboration among multiple agents with strategic consensus checking for efficiency. Our extensive experiments demonstrate that CollabEval consistently outperforms single-LLM approaches across multiple dimensions while maintaining robust performance even when individual models struggle. The framework provides comprehensive support for various evaluation criteria while ensuring efficiency through its collaborative design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
