Recent advancements in large language models (LLMs) have given rise to the LLM-as-a-judge paradigm, showcasing their potential to deliver human-like judgments. However, in the field of machine translation (MT) evaluation, current LLM-as-a-judge methods fall short of learned automatic metrics. In this paper, we propose Multidimensional Multi-Agent Debate (M-MAD), a systematic LLM-based multi-agent framework for advanced LLM-as-a-judge MT evaluation. Our findings demonstrate that M-MAD achieves significant advancements by (1) decoupling heuristic MQM criteria into distinct evaluation dimensions for fine-grained assessments; (2) employing multi-agent debates to harness the collaborative reasoning capabilities of LLMs; (3) synthesizing dimension-specific results into a final evaluation judgment to ensure robust and reliable outcomes. Comprehensive experiments show that M-MAD not only outperforms all existing LLM-as-a-judge methods but also competes with state-of-the-art reference-based automatic metrics, even when powered by a suboptimal model like GPT-4o mini. Detailed ablations and analysis highlight the superiority of our framework design, offering a fresh perspective for LLM-as-a-judge paradigm. Our code and data are publicly available at https://github.com/SU-JIAYUAN/M-MAD.
MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100085, China(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China(中国科学院大学网络安全学院)
Gradual Vigilance and Interval Communication: Enhancing Value Alignment in Multi-Agent Debates
National Engineering Research Center of Educational Big Data, Central China Normal University(华中师范大学教育大数据国家工程研究中心);Faculty of Artificial Intelligence in Education, Central China Normal University(华中师范大学人工智能教育学部)
MADRA: Multi-Agent Debate for Risk-Aware Embodied Planning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);University of Chinese Academy of Sciences, Nanjing(中国科学院大学南京校区)
Breaking the Martingale Curse: Multi-Agent Debate via Asymmetric Cognitive Potential Energy
Mohamed bin Zayed University of Artificial Intelligence(莫扎德·本·扎耶德人工智能大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学国际关系学院);Harvard University(哈佛大学)
CortexDebate: Debating Sparsely and Equally for Multi-Agent Debate
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);College of Artificial Intelligence, Nanjing Agricultural University(南京农业大学人工智能学院);School of Automation and Intelligent Sensing, Shanghai Jiao Tong University(上海交通大学自动化与智能感知学院)
Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
Boston Children’s Hospital, Harvard Medical School(波士顿儿童医院,哈佛医学院);Harvard School of Engineering and Applied Sciences, Harvard College(哈佛工程与应用科学学院,哈佛学院)
DynaDebate: Breaking Homogeneity in Multi-Agent Debate with Dynamic Path Generation
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);North Automatic Control Technology Institute(北自动控制技术研究所);Shenzhen Institute for Advanced Study, UESTC(深圳先进研究 institute, 电子科技大学)
MIND: Multi-agent inference for negotiation dialogue in travel planning
School of Mechanical Engineering, Sungkyunkwan University(成均馆大学机械工程学院);Department of Applied Artificial Intelligence, Sungkyunkwan University(成均馆大学应用人工智能系);Department of Software, Sungkyunkwan University(成均馆大学软件系)
Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoning
University of Cambridge(剑桥大学);Sorbonne Université(索邦大学);University of Science and Technology of China(中国科学技术大学);Beihang University(北航大学);Nanyang Technological University(南洋理工大学)
Beyond Solo and Consistency: Vindicating Multi-Agent Debate via Conditional Progressive Pruning
Rutgers University, New Brunswick(罗格斯大学新布朗斯维克分校);University of Cambridge(剑桥大学);Tsinghua University(清华大学);Harvard University(哈佛大学);Case Western Reserve University(凯斯西储大学);University of California San Diego(加利福尼亚大学圣迭戈分校);Carnegie Mellon University(卡内基梅隆大学)