Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) how good LLMs are by themselves in generating and validating simple plans in commonsense planning tasks (of the type that humans are generally quite good at) and (2) how good LLMs are in being a source of heuristic guidance for other agents--either AI planners or human planners--in their planning tasks. To investigate these questions in a systematic rather than anecdotal manner, we start by developing a benchmark suite based on the kinds of domains employed in the International Planning Competition. On this benchmark, we evaluate LLMs in three modes: autonomous, heuristic and human-in-the-loop. Our results show that LLM's ability to autonomously generate executable plans is quite meager, averaging only about 3% success rate. The heuristic and human-in-the-loop modes show slightly more promise. In addition to these results, we also make our benchmark and evaluation tools available to support investigations by research community.
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
Beijing University of Posts and Telecommunications（北京邮电大学）；The Hong Kong University of Science and Technology（香港科技大学）；Beihang University（北京航空航天大学）；University of the Chinese Academy of Sciences（中国科学院大学）；ETH Zürich（苏黎世联邦理工学院）；The Hong Kong Polytechnic University（香港理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai AI Lab（上海人工智能实验室）
Non-myopic Generation of Language Models for Reasoning and Planning
The University of Hong Kong（香港大学）；Peking University（北京大学）；Zhejiang University（浙江大学）；Westlake University（西湖大学）；The Hong Kong University of Science and Technology（香港科技大学）
Pet-Bench: Benchmarking the Abilities of Large Language Models as E-Pets in Social Network Services
Fudan University（复旦大学）；Xiaohongshu Inc.（小红书公司）；Beihang University（北航）；Zhejiang University（浙江大学）；Singapore University of Technology and Design（新加坡科技设计大学）
GAUSS: Benchmarking Structured Mathematical Skills for Large Language Models
Hyperbolic Labs；California Institute of Technology（加州理工学院）；University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；Nvidia（英伟达）；University of Washington（华盛顿大学）；University of Hong Kong（香港大学）
KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models
Huawei Poisson Laboratory（华为泊松实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Hong Kong University of Science and Technology（香港科技大学）
Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task
Stanford University（斯坦福大学）；Australian National University（澳大利亚国立大学）；University of St. Andrews（圣安德鲁斯大学）；University of Cambridge（剑桥大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models
University of Chicago（芝加哥大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；Syracuse University（雪城大学）；WeChat AI（微信人工智能）；New Jersey Institute of Technology（新泽西理工学院）
RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际 Graduate School）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Yinwang Intelligent Technology Co., Ltd.（亿纬智能科技有限公司）；Huawei’s 2012 Lab（华为2012实验室）
Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities
College of Computer Science；Technology, National University of Defense Technology, Changsha, Hunan, China；State Key Laboratory of Complex \& Critical Software Environment, Changsha, Hunan, China；National Key Laboratory of Parallel；School of Humanities；Social Sciences, School of Public Administration, Beihang University, Beijing, China
GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
Olin College of Engineering（奥林工程学院）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Oxford（牛津大学）；University College London（伦敦大学学院）；ShanghaiTech University（上海科技大学）；Columbia University（哥伦比亚大学）；University of Manchester（曼彻斯特大学）
NUMINA: A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities
School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学先进技术学院）；Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of Deep Perception Technology, JITRI（深度感知技术研究所）
