LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
Beijing University of Posts and Telecommunications(北京邮电大学);Paradigm Inc.(第四范式公司);Tsinghua University(清华大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Recent advancements in large language models (LLMs) have revealed their potential for achieving autonomous agents possessing human-level intelligence. However, existing benchmarks for evaluating LLM Agents either use static datasets, potentially leading to data leakage or focus only on single-agent scenarios, overlooking the complexities of multi-agent interactions. There is a lack of a benchmark that evaluates the diverse capabilities of LLM agents in multi-agent, dynamic environments. To this end, we introduce LLMArena, a novel and easily extensible framework for evaluating the diverse capabilities of LLM in multi-agent dynamic environments. LLMArena encompasses seven distinct gaming environments, employing Trueskill scoring to assess crucial abilities in LLM agents, including spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We conduct an extensive experiment and human evaluation among different sizes and types of LLMs, showing that LLMs still have a significant journey ahead in their development towards becoming fully autonomous agents, especially in opponent modeling and team collaboration. We hope LLMArena could guide future research towards enhancing these capabilities in LLMs, ultimately leading to more sophisticated and practical applications in dynamic, multi-agent settings. The code and data will be available.
Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain
Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification
University of Warwick(沃里克大学);Queen Mary University of London(伦敦大学女王学院);University of Bologna(博洛尼亚大学);King’s College London(伦敦大学国王学院);The Alan Turing Institute(艾伦·图灵研究所)
EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Beijing Future Brain Education Technology Co., Ltd.(北京未来脑教育科技有限公司);The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);Guangxi Zhuang Autonomous Region Big Data Research Institute(广西壮族自治区大数据研究院)
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalism
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);Institute for Advanced Algorithms Research, China(中国高等算法研究院);Northeastern University, China(中国东北大学);State Key Laboratory of Media Convergence Production Technology and Systems, China(中国媒体融合生产技术与系统国家重点实验室)
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Tencent Robotics X(腾讯Robotics X实验室);University of Toronto(多伦多大学);Zhili College, Tsinghua University(清华大学志道书院)
MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University(沃克生物医学工程部门,佐治亚理工学院和埃默里大学);College of Future of Technology, Peking University(未来技术学院,北京大学);School of Architecture, Tsinghua University(建筑学院,清华大学);School of Electrical and Computer Engineering, Georgia Institute of Technology(电气与计算机工程学院,佐治亚理工学院);School of Computer Science, Georgia Institute of Technology(计算机科学学院,佐治亚理工学院)
Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
GeoDS Lab, Department of Geography, University of Wisconsin-Madison, USA(威斯康星大学麦迪逊分校地理系GeoDS实验室);Department of Geography and Regional Research, University of Vienna, Austria(维也纳大学地理与区域研究系);Department of Geography, University of California-Santa Barbara, USA(加州大学圣塔芭芭拉分校地理系)
Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);School of Computer Science, University of Sheffield(谢菲尔德大学计算机科学学院);Idiap Research Institute(Idiap研究机构);National Biomarker Centre, CRUK Manchester Institute(国家生物标志物中心、CRUK曼彻斯特研究所)
LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues
Faculty of Information Technology, Monash University, Australia(墨尔本大学信息技术学院,澳大利亚);School of Information Technology, Monash University Malaysia(墨尔本大学马来西亚分校信息学院);School of Business, Monash University Malaysia(墨尔本大学马来西亚分校商学院);Faculty of Law, Monash University, Australia(墨尔本大学法学院,澳大利亚)
Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes