Large Language Models (LLMs) based agent systems have made great strides in real-world applications beyond traditional NLP tasks. This paper proposes a new LLM-based Multi-Agent System (LLM-MAS) benchmark, Collab-Overcooked, built on the popular Overcooked-AI game with more applicable and challenging tasks in interactive environments. Collab-Overcooked extends existing benchmarks in two novel ways. First, it provides a multi-agent framework supporting diverse tasks and objectives and encourages collaboration through natural language communication. Second, it introduces a spectrum of process-oriented evaluation metrics to assess the fine-grained collaboration capabilities of different LLM agents, a dimension often overlooked in prior work. We conduct extensive experiments with 13 popular LLMs and show that, while the LLMs exhibit a strong ability in goal interpretation, there are significant shortcomings in active collaboration and continuous adaptation, which are critical for efficiently fulfilling complex tasks. Notably, we highlight the strengths and weaknesses of LLM-MAS and provide insights for improving and evaluating LLM-MAS on a unified and open-source benchmark. The environments, 30 open-ended tasks, and the evaluation package are publicly available at https://github.com/YusaeMeow/Collab-Overcooked.
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
Shanghai Institute of AI for Education(上海人工智能教育研究院);School of Computer Science(计算机科学学院);East China Normal University(东华大学);Tencent Inc.(腾讯公司);Shanghai Innovation Institute(上海创新研究院)
CoWork-X: Experience-Optimized Co-Evolution for Multi-Agent Collaboration System
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)科学与工程学院);School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)人工智能学院);Tsinghua University(清华大学);AutoGame Research(AutoGame研究);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院)
ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The University of Sydney(悉尼大学);The Chinese University of Hong Kong(香港中文大学)
MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks
National Engineering Research Center for Software Engineering, Peking University(北京大学国家软件工程研究中心);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院);Centre for Medical Informatics, The University of Edinburgh(爱丁堡大学医学信息学中心);ETH Zurich(苏黎世联邦理工学院);Health Data Research UK(英国健康数据研究所);Key Laboratory of High Confidence Software Technologies, Ministry of Education(教育部高可信软件技术重点实验室)
Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration
Tsinghua University(清华大学);Northwestern Polytechnical University(西北工业大学);Institute of Artificial Intelligence (TeleAI)(人工智能研究所);China Telecom(中国电信);Zhejiang University(浙江大学);Shenzhen Research Institute of Northwestern Polytechnical University(西北工业大学深圳研究院)
Leverhulme Centre for the Future of Intelligence, University of Cambridge(未来智能中心,剑桥大学);Institute for Human-Centered AI, Helmholtz Zentrum Munich(人本人工智能研究所,慕尼黑赫尔姆霍尔茨中心);VRAIN, Universitat Politècnica de València(VRAIN,瓦伦西亚理工大学)
Interchangeable in Role, Not in History: Replacing Members of Formed LLM Teams
China Agricultural University(中国农业大学);Tianjin University of Finance and Economics(天津财经大学);Jilin University(吉林大学);Tianjin University of Science and Technology(天津科技大学)
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);New York University Abu Dhabi(纽约大学阿布扎比分校);IBM Research AI(IBM研究院人工智能部门);Qassim University(卡西姆大学)
D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios
TJUNLP Lab, School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院TJUNLP实验室);The International Joint Institute of Tianjin University(天津大学国际联合研究院)
SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training
McGill University(麦吉尔大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);Institute of Health Sciences Education, McGill University(麦吉尔大学健康科学教育研究所);School of Social Work, McGill University(麦吉尔大学社会工作学院);Montreal Children’s Hospital, McGill University Health Centre(麦吉尔大学健康中心蒙特利尔儿童医院);Steinberg Centre for Simulation and Interactive Learning, McGill University(麦吉尔大学斯坦伯格模拟与互动学习中心);Linarite AI(利纳里特人工智能公司)
Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models
School of Artificial Intelligence, Beihang University, Beijing, China(北京航空航天大学人工智能学院);KoGuan School of Law, Shanghai Jiao Tong University, Shanghai, China(上海交通大学KoGuan法学院);School of Criminal Law, China University of Political Science and Law, Beijing, China(中国政法大学刑事法律学院)
RefactorCoderQA: Benchmarking LLMs for Multi-Domain Coding Question Solutions in Cloud and Edge Deployment
York University and Algoma University(约克大学和阿尔戈马大学);Algoma University(阿尔戈马大学);Department of Systems and Computer Engineering, Carleton University(系统与计算机工程系,卡尔顿大学);Department of Math and Computer Science, Brandon University(数学与计算机科学系,布兰登大学)
A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System
Polytechnic University of Bari(巴里理工大学);Peoples' Friendship University of Russia (RUDN University)(俄罗斯人民友谊大学(RUDN大学));Joint Institute for Nuclear Research(联合核子研究所);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);University of Skövde(舍夫德大学)
Can Multimodal LLMs See Science Instruction? Benchmarking Pedagogical Reasoning in K-12 Classroom Videos
Department of Computer Science, Drexel University(德雷塞尔大学计算机科学系);Department of Teaching and Learning, Washington State University(华盛顿州立大学教学与学习系);College of Chemistry, Beijing Normal University(北京师范大学化学学院);Department of Computer Science, University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校计算机科学系);Department of Data Science, City University of Hong Kong(香港城市大学数据科学系)
ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));Microsoft(微软);ELLIS Institute Tübingen and MPI for Intelligent Systems(图宾根ELLIS研究所和智能系统研究所);Tübingen AI Center(图宾根人工智能中心)