Multi-Agent-as-Judge：使基于 LLM 智能体的自动化评估与多维人类评估保持一致
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation
AI总结：
本文提出 MAJ-EVAL 多智能体评审框架，从文档自动构建多维评估者角色并通过组内辩论生成反馈，在教育和医学领域提升了与人类专家评分的一致性。
摘要：
几乎所有人类工作都是协作性的；因此，对现实世界 NLP 应用的评估通常需要多个维度，以契合不同的人类视角。由于真实人类评估者资源往往稀缺且成本高昂，新兴的“LLM-as-a-judge”范式为利用 LLM 智能体可信地模拟人类评估者提供了一种有前景的方法。然而，迄今为止，现有 LLM-as-a-judge 方法面临两个局限：智能体的角色描述通常是任意设计的，而且这些框架不能泛化到其他任务。为应对这些挑战，我们提出 MAJ-EVAL，这是一个 Multi-Agent-as-Judge 评估框架，能够从相关文本文档（例如研究论文）中自动构建具有不同维度的多个评估者角色，用这些角色实例化 LLM 智能体，并通过多智能体组内辩论生成多维反馈。我们在教育和医学领域的评估实验表明，与传统自动化评估指标和现有 LLM-as-a-judge 方法相比，MAJ-EVAL 能够生成与人类专家评分更一致的评估结果。
英文摘要：
Nearly all human work is collaborative; thus, the evaluation of real-world NLP applications often requires multiple dimensions that align with diverse human perspectives. As real human evaluator resources are often scarce and costly, the emerging "LLM-as-a-judge" paradigm sheds light on a promising approach to leverage LLM agents to believably simulate human evaluators. Yet, to date, existing LLM-as-a-judge approaches face two limitations: persona descriptions of agents are often arbitrarily designed, and the frameworks are not generalizable to other tasks. To address these challenges, we propose MAJ-EVAL, a Multi-Agent-as-Judge evaluation framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents (e.g., research papers), instantiate LLM agents with the personas, and engage in-group debates with multi-agents to Generate multi-dimensional feedback. Our evaluation experiments in both the educational and medical domains demonstrate that MAJ-EVAL can generate evaluation results that better align with human experts' ratings compared with conventional automated evaluation metrics and existing LLM-as-a-judge methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
