主流大语言模型已在众多领域展现出人类水平的性能,甚至使衍生出的智能体能够模拟人类与社会互动。尽管实践工作已经证实将语言智能体落地到沙盒模拟或具身模拟器中的可行性,但当前的社会智能基准要么停留在语言层面,要么使用主观指标。为追求更真实、客观的评估,我们提出 Social Tasks in Sandbox Simulation(STSS)基准,通过在多智能体模拟中审查目标达成情况,在动作层面客观评估语言智能体。此外,我们抽样对话场景以构建语言层面基准,从而提供经济审慎的初步评估,并与主流基准保持一致。为衡量智能体架构的重要性,我们实现了一个目标驱动规划(TDP)模块,作为现有智能体的附加组件。我们的评估结果表明,STSS 基准对最先进的语言智能体具有挑战性。此外,它能有效区分不同的语言智能体,表明其适合作为评估语言模型和智能体架构的基准。
英文摘要:
Prominent large language models have exhibited human-level performance in many domains, even enabling the derived agents to simulate human and social interactions. While practical works have substantiated the practicability of grounding language agents in sandbox simulation or embodied simulators, current social intelligence benchmarks either stay at the language level or use subjective metrics. In pursuit of a more realistic and objective evaluation, we introduce the Social Tasks in Sandbox Simulation (STSS) benchmark, which assesses language agents \textbf{objectively} at the \textbf{action level} by scrutinizing the goal achievements within the multi-agent simulation. Additionally, we sample conversation scenarios to build a language-level benchmark to provide an economically prudent preliminary evaluation and align with prevailing benchmarks. To gauge the significance of agent architecture, we implement a target-driven planning (TDP) module as an adjunct to the existing agent. Our evaluative findings highlight that the STSS benchmark is challenging for state-of-the-art language agents. Furthermore, it effectively discriminates between distinct language agents, suggesting its usefulness as a benchmark for evaluating both language models and agent architectures.
OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence
School of Software Technology, Zhejiang University(浙江大学软件学院);Laboratory for Statistical Monitoring and Intelligent Governance of Common Prosperity, Zhejiang Gongshang University(浙江工商大学共同富裕统计监测与智能治理实验室);Tongyi Lab, Alibaba Group(阿里集团通义实验室);Alibaba Cloud(阿里云)
EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models
Shenzhen University(深圳大学);Guangdong Laboratory of Artificial Intelligence(广东人工智能与数字经济实验室);Auckland University of Technology(奥克兰理工大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);SKL-IOTSC, CIS, University of Macau(澳门科学技术大学SKL-IOTSC、CIS)
SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence
Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);University of Sydney(悉尼大学);Fudan University(复旦大学);Tsinghua University(清华大学);Beihang University(北航大学)
Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence
Technology Innovation Institute(技术创新研究院);University of Oslo(奥斯陆大学);Eötvös Loránd University(厄特沃什·罗兰大学);University of Guelma(盖勒马大学);The University of Manchester(曼彻斯特大学);Khalifa University(哈利法大学);Cyentific AS(赛恩蒂菲克公司)
GAUSS: Benchmarking Structured Mathematical Skills for Large Language Models
Hyperbolic Labs;California Institute of Technology(加州理工学院);University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);Nvidia(英伟达);University of Washington(华盛顿大学);University of Hong Kong(香港大学)
Rationality Check! Benchmarking the Rationality of Large Language Models
Department of Electronic Engineering, BNRist, Tsinghua University(电子工程系、BNRist、清华大学);Department of Sociology, University of Chicago(社会学系、芝加哥大学);Santa Fe Institute(圣菲研究所)
MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Shanghai Jiao Tong University(上海交通大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Huawei Noah’s Ark Lab(华为诺亚实验室)
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation
School of Computer Science and Engineering, The Hebrew University of Jerusalem(计算机科学与工程学院,特拉维夫大学);IBM Research(IBM研究院);The Allen Institute for AI (AI2)(人工智能研究所)
CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges
Southern University of Science and Technology(南方科技大学);Tsinghua University(清华大学);Peking University(北京大学);University of Bristol(布里斯托大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Xi’an Jiaotong University(西安交通大学)
Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Institute of Technology(北京理工大学);Xi’an Jiaotong University(西安交通大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
SpaceDG: Benchmarking Spatial Intelligence under Visual Degradation
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Electronic Science and Technology of China(电子科技大学);Chongqing University(重庆大学);The University of Tokyo(东京大学);Beihang University(北航);Northwestern Polytechnical University(西北工业大学)
Mapping global dynamics of benchmark creation and saturation in artificial intelligence
Medical University of Vienna(维也纳医科大学);ITTM S.A.—Information Technology for Translational Medicine(ITTM S.A.——转化医学信息技术公司);University of Oxford(牛津大学);Future of Humanity Institute, University of Oxford(牛津大学人类未来研究所)