arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.20127cs.CLcs.AI
收藏

M-MAD:用于先进机器翻译评估的多维度多智能体辩论

M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation

Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学)

Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, Zuozhu Liu

AI总结:

本文提出多智能体框架 M-MAD,通过维度解耦、多智能体辩论与结果综合改进机器翻译评估,在 GPT-4o mini 上即可超越现有 LLM 评判方法并媲美顶尖参考型自动指标。

摘要:

大语言模型(LLM)的最新进展催生了 LLM-as-a-judge(大语言模型作为评判者)范式,展现出其提供类人判断的潜力。然而,在机器翻译(MT)评估领域,当前的 LLM-as-a-judge 方法仍不及基于学习的自动指标。本文提出多维度多智能体辩论(Multidimensional Multi-Agent Debate,M-MAD),这是一个系统化的、基于 LLM 的多智能体框架,用于先进的 LLM-as-a-judge 机器翻译评估。研究结果表明,M-MAD 通过以下方式取得显著进展:(1)将启发式 MQM 标准解耦为不同的评估维度,以进行细粒度评估;(2)采用多智能体辩论来利用 LLM 的协作推理能力;(3)将各维度特定结果综合为最终评估判断,以确保结果稳健可靠。综合实验表明,M-MAD 不仅优于所有现有 LLM-as-a-judge 方法,而且即使由 GPT-4o mini 这类次优模型驱动,也能与最先进的基于参考译文的自动指标相竞争。详细的消融实验和分析突显了该框架设计的优越性,为 LLM-as-a-judge 范式提供了新视角。代码和数据公开发布于 https://github.com/SU-JIAYUAN/M-MAD。

英文摘要:

Recent advancements in large language models (LLMs) have given rise to the LLM-as-a-judge paradigm, showcasing their potential to deliver human-like judgments. However, in the field of machine translation (MT) evaluation, current LLM-as-a-judge methods fall short of learned automatic metrics. In this paper, we propose Multidimensional Multi-Agent Debate (M-MAD), a systematic LLM-based multi-agent framework for advanced LLM-as-a-judge MT evaluation. Our findings demonstrate that M-MAD achieves significant advancements by (1) decoupling heuristic MQM criteria into distinct evaluation dimensions for fine-grained assessments; (2) employing multi-agent debates to harness the collaborative reasoning capabilities of LLMs; (3) synthesizing dimension-specific results into a final evaluation judgment to ensure robust and reliable outcomes. Comprehensive experiments show that M-MAD not only outperforms all existing LLM-as-a-judge methods but also competes with state-of-the-art reference-based automatic metrics, even when powered by a suboptimal model like GPT-4o mini. Detailed ablations and analysis highlight the superiority of our framework design, offering a fresh perspective for LLM-as-a-judge paradigm. Our code and data are publicly available at https://github.com/SU-JIAYUAN/M-MAD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑