arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.02867cs.CLcs.AI

Multi-Agent LLM Judge：用于评估自然语言生成应用的自动个性化 LLM 评判器设计

Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications

Amadeus SAS

Hongliu Cao, Ilias Driouich, Robin Singh, Eoin Thomas

备注：
Presented at SophiaSummit2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有 LLM-as-a-judge 框架泛化性差且与人类判断相关性低的问题，提出动态多智能体系统自动设计个性化 LLM 评判器，通过迭代优化评估提示平衡任务需求与人类感知，显著提升了评估准确率与对齐度。

摘要：

大型语言模型（LLMs）在各个领域展现出了出色的性能，但仍然面临领域知识不足、偏见和幻觉等挑战。这凸显了需要稳健的评估方法来准确评估基于 LLM 的应用。依赖词重叠或文本嵌入的传统评估方法，无法捕捉评估动态、开放式文本生成所需的细微语义信息。近期研究探索利用 LLM 模仿人类推理和决策过程进行评估，即 LLM-as-a-judge 框架。然而，这些现有框架存在两个显著局限。首先，它们缺乏适应不同文本风格（包括各种答案和真实答案风格）的灵活性，从而降低了泛化性能。其次，这些框架产生的评估分数通常存在偏差且难以解释，与人类判断的相关性较低。为了应对这些挑战，我们提出了一种新颖的动态多智能体系统，能够为各种自然语言生成应用自动设计个性化的 LLM 评判器。该系统迭代地优化评估提示，并平衡下游任务的自适应要求与人类感知对齐之间的权衡。实验结果表明，所提出的多智能体 LLM Judge 框架不仅提高了评估准确率，而且产生的评估分数更符合人类感知。

英文摘要：

Large Language Models (LLMs) have demonstrated impressive performance across diverse domains, yet they still encounter challenges such as insufficient domain-specific knowledge, biases, and hallucinations. This underscores the need for robust evaluation methodologies to accurately assess LLM-based applications. Traditional evaluation methods, which rely on word overlap or text embeddings, are inadequate for capturing the nuanced semantic information necessary to evaluate dynamic, open-ended text generation. Recent research has explored leveraging LLMs to mimic human reasoning and decision-making processes for evaluation purposes known as LLM-as-a-judge framework. However, these existing frameworks have two significant limitations. First, they lack the flexibility to adapt to different text styles, including various answer and ground truth styles, thereby reducing their generalization performance. Second, the evaluation scores produced by these frameworks are often skewed and hard to interpret, showing a low correlation with human judgment. To address these challenges, we propose a novel dynamic multi-agent system that automatically designs personalized LLM judges for various natural language generation applications. This system iteratively refines evaluation prompts and balances the trade-off between the adaptive requirements of downstream tasks and the alignment with human perception. Our experimental results show that the proposed multi-agent LLM Judge framework not only enhances evaluation accuracy compared to existing methods but also produces evaluation scores that better align with human perception.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑