In recent advancements, large language models (LLMs) have exhibited proficiency in code generation and chain-of-thought reasoning, laying the groundwork for tackling automatic formal planning tasks. This study evaluates the potential of LLMs to understand and generate Planning Domain Definition Language (PDDL), an essential representation in artificial intelligence planning. We conduct an extensive analysis across 20 distinct models spanning 7 major LLM families, both commercial and open-source. Our comprehensive evaluation sheds light on the zero-shot LLM capabilities of parsing, generating, and reasoning with PDDL. Our findings indicate that while some models demonstrate notable effectiveness in handling PDDL, others pose limitations in more complex scenarios requiring nuanced planning knowledge. These results highlight the promise and current limitations of LLMs in formal planning tasks, offering insights into their application and guiding future efforts in AI-driven planning paradigms.
Generating Symbolic World Models via Test-time Scaling of Large Language Models
Westlake University（西湖大学）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Max Planck Institute for Intelligent Systems, Tübingen（智能系统马克斯·普朗克研究所（图宾根））；Environmental Systems Research Institute, Inc.（环境系统研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；SEED, Bytedance（字节跳动SEED团队）；The Hong Kong University of Science and Technology（香港科技大学）
The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Cogitai, Inc.（科吉泰公司）；SUNY Binghamton（纽约州立大学宾汉姆顿分校）
Robots need task planning algorithms to sequence actions toward accomplishing goals that are impossible through individual actions. Off-the-shelf task planners can be used by intelligent robotics practitioners to solve a variety of planning problems. However, many different planners exist, each with different strengths and weaknesses, and there are no general rules for which planner would be best to apply to a given problem. In this article, we empirically compare the performance of state-of-the-art planners that use either the Planning Domain Description Language (PDDL), or Answer Set Program…
From Capability Models to Automated Planning: An AAS-Native Approach for Automatic PDDL Generation
Institute of Automation Technology, Helmut Schmidt University Hamburg（海德堡-施密特大学汉堡自动化技术研究所）；Chair of Automation Technology, Ruhr University Bochum（博尔塔伦大学博德姆自动化技术教授团）
In recent years research in the planning community has moved increasingly toward s application of planners to realistic problems involving both time and many typ es of resources. For example, interest in planning demonstrated by the space res earch community has inspired work in observation scheduling, planetary rover ex ploration and spacecraft control domains. Other temporal and resource-intensive domains including logistics planning, plant control and manufacturing have also helped to focus the community on the modelling and reasoning issues that must be confronted to make planning technolo…
Planning is concerned with the automated solution of action sequencing problems described in declarative languages giving the action preconditions and effects. One important application area for such technology is the creation of new processes in Business Process Management (BPM), which is essential in an ever more dynamic business environment. A major obstacle for the application of Planning in this area lies in the modeling. Obtaining a suitable model to plan with -- ideally a description in PDDL, the most commonly used planning language -- is often prohibitively complicated and/or costly. O…
Institute of Cognitive Science and Technologies, National Research Council（国家研究委员会认知科学与技术研究所）
In symbolic planning systems, the knowledge on the domain is commonly provided by an expert. Recently, an automatic abstraction procedure has been proposed in the literature to create a Planning Domain Definition Language (PDDL) representation, which is the most widely used input format for most off-the-shelf automated planners, starting from `options', a data structure used to represent actions within the hierarchical reinforcement learning framework. We propose an architecture that potentially removes the need for human intervention. In particular, the architecture first acquires options in …
Contextual word representation models have shown massive improvements on a multitude of NLP tasks, yet their word sense disambiguation capabilities remain poorly explained. To address this gap, we assess whether contextual word representations extracted from deep pretrained language models create distinguishable representations for different senses of a given word. We analyze the representation geometry and find that most layers of deep pretrained language models create highly anisotropic representations, pointing towards the existence of representation degeneration problem in contextual word …
Recent years pretrained language models (PLMs) hit a success on several downstream tasks, showing their power on modeling language. To better understand and leverage what PLMs have learned, several techniques have emerged to explore syntactic structures entailed by PLMs. However, few efforts have been made to explore grounding capabilities of PLMs, which are also essential. In this paper, we highlight the ability of PLMs to discover which token should be grounded to which concept, if combined with our proposed erasing-then-awakening approach. Empirical studies on four datasets demonstrate that…
Code Simulation Challenges for Large Language Models
University of Oxford（牛津大学）；Technical University of Munich（慕尼黑工业大学）；University of Salerno（萨勒诺大学）；University of Bologna（博洛尼亚大学）；University of Leeds（利兹大学）
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University（浙江大学 ZIP 实验室）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人教育部重点实验室）
Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation
The Hong Kong Polytechnic University（香港理工大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Great Bay University（大湾区大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Analogies play a central role in human commonsense reasoning. The ability to recognize analogies such as "eye is to seeing what ear is to hearing", sometimes referred to as analogical proportions, shape how we structure knowledge and understand language. Surprisingly, however, the task of identifying such analogies has not yet received much attention in the language model era. In this paper, we analyze the capabilities of transformer-based language models on this unsupervised task, using benchmarks obtained from educational settings, as well as more commonly used datasets. We find that off-the…
