Large Language Models (LLMs) are rapidly evolving and impacting various fields, necessitating the development of effective methods to evaluate and compare their performance. Most current approaches for performance evaluation are either based on fixed, domain-specific questions that lack the flexibility required in many real-world applications, or rely on human input, making them unscalable. To address these issues, we propose an automated benchmarking framework based on debates between LLMs, judged by another LLM. This method assesses not only domain knowledge, but also skills such as argumentative reasoning and inconsistency recognition. We evaluate the performance of various state-of-the-art LLMs using the debate framework and achieve rankings that align closely with popular rankings based on human input, eliminating the need for costly human crowdsourcing.
Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation
School of Computer Science, University College Dublin(都柏林大学计算机科学学院);School of Politics and International Relations, University College Dublin(都柏林大学政治与国际关系学院);Department of Informatics, King’s College London(伦敦国王学院信息学院)
Debate-to-Detect: Reformulating Misinformation Detection as a Real-World Debate with Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院)
ARMOR-MAD: Adaptive Routing for Heterogeneous Multi-Agent Debate in Large Language Model Reasoning
School of Cyber Science and Technology, University of Science and Technology of China(中国科学技术大学网络空间安全学院);School of Artificial Intelligence, Shenzhen Technology University(深圳技术大学人工智能学院)
Debate-on-Graph: Reliable and Adaptive Reasoning of Large Language Model on Uncertain Knowledge Graph
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University)(教育部新一代人工智能技术及其跨学科应用重点实验室(东南大学))
Debating Truth: Debate-driven Claim Verification with Multiple Large Language Model Agents
Department of Interactive Media, Hong Kong Baptist University(香港浸会大学互动媒体系);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Beijing Normal-Hong Kong Baptist University(北京师范大学-香港浸会大学联合国际学院);Shenzhen Institute of Advanced Technology(深圳先进技术研究院)
DS@GT at Touché: Large Language Models for Retrieval-Augmented Debate
Georgia Institute of Technology(佐治亚理工学院);Peoples' Friendship University of Russia (RUDN University)(俄罗斯人民友谊大学);Joint Institute for Nuclear Research(联合核研究所);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);University of Skövde(斯德哥尔摩大学)
Analysing Personal Attacks in U.S. Presidential Debates
School of Computer Science and Engineering, University of New South Wales(计算机科学与工程学院,新南威尔士大学);School of Mathematics and Statistics, University of New South Wales(数学与统计学学院,新南威尔士大学)
Debate on Graph: a Flexible and Reliable Reasoning Framework for Large Language Models
Xi’an Jiaotong University(西安交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Northwestern Polytechnical University(西北工业大学);Zhejiang Createlink Technology(浙江创联科技);Shandong University(山东大学)
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation
School of Computer Science and Engineering, The Hebrew University of Jerusalem(计算机科学与工程学院,特拉维夫大学);IBM Research(IBM研究院);The Allen Institute for AI (AI2)(人工智能研究所)
When to Trust Context: Self-Reflective Debates for Context Reliability
Brown University(布朗大学);Stanford University(斯坦福大学);University of New South Wales(新南威尔士大学);Xi’an University of Electronic Science and Technology(西安电子科技大学);University of Chicago(芝加哥大学)
Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debate
Institutes of Innovation for Future Society(未来社会创新研究所);Nagoya University(名古屋大学);Department of Micro-Nano Mechanical Science and Engineering(微纳米机械科学与工程系)