In generative commonsense reasoning tasks such as CommonGen, generative large language models (LLMs) compose sentences that include all given concepts. However, when focusing on instruction-following capabilities, if a prompt specifies a concept order, LLMs must generate sentences that adhere to the specified order. To address this, we propose Ordered CommonGen, a benchmark designed to evaluate the compositional generalization and instruction-following abilities of LLMs. This benchmark measures ordered coverage to assess whether concepts are generated in the specified order, enabling a simultaneous evaluation of both abilities. We conducted a comprehensive analysis using 36 LLMs and found that, while LLMs generally understand the intent of instructions, biases toward specific concept order patterns often lead to low-diversity outputs or identical results even when the concept order is altered. Moreover, even the most instruction-compliant LLM achieved only about 75% ordered coverage, highlighting the need for improvements in both instruction-following and compositional generalization capabilities.
Standard architectures used in instruction following often struggle on novel compositions of subgoals (e.g. navigating to landmarks or picking up objects) observed during training. We propose a modular architecture for following natural language instructions that describe sequences of diverse subgoals. In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type. A sequence of modules to execute is chosen by learning to segment the instructions and predicting a subgoal type for each segment. When compared to standard, non-modular sequence-to-sequenc…
This paper describes an alignment-based model for interpreting natural language instructions in context. We approach instruction following as a search over plans, scoring sequences of actions conditioned on structured observations of text and the environment. By explicitly modeling both the low-level compositional structure of individual actions and the high-level structure of full plans, we are able to learn both grounded representations of sentence meaning and pragmatic constraints on interpretation. To demonstrate the model's flexibility, we apply it to a diverse set of benchmark tasks. On …
Towards Compositional Generalization of LLMs via Skill Taxonomy Guided Data Synthesis
State Key Laboratory of Complex & Critical Software Environment, Beihang University（复杂与关键软件环境国家重点实验室，北京航空航天大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Institute of Technology（北京理工大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
Compositional Instruction Following with Language Models and Reinforcement Learning
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of the Witwatersrand（威特沃特斯兰德大学）；Arizona State University（亚利桑那州立大学）；Georgia Institute of Technology（佐治亚理工学院）
Revisiting Prompt Sensitivity in Large Language Models for Text Classification: The Role of Prompt Underspecification
Kempelen Institute of Intelligent Technologies（智能技术研究所）；Faculty of Information Technology, Brno University of Technology（信息科技学院）；Faculty of Informatics, Masaryk University（信息学院）
Compositional Steering of Large Language Models with Steering Tokens
NEC Laboratories Europe（NEC欧洲实验室）；University of Edinburgh（爱丁堡大学）；Center for Artificial Intelligence and Data Science（人工智能与数据科学中心）；University of Würzburg（乌尔姆大学）；CAIR, Ss. Cyril and Methodius University of Skopje（CAIR，斯·西里尔和方法ius大学）
Human intelligence exhibits compositional generalization (i.e., the capacity to understand and produce unseen combinations of seen components), but current neural seq2seq models lack such ability. In this paper, we revisit iterative back-translation, a simple yet effective semi-supervised method, to investigate whether and how it can improve compositional generalization. In this work: (1) We first empirically show that iterative back-translation substantially improves the performance on compositional generalization benchmarks (CFQ and SCAN). (2) To understand why iterative back-translation is …
Evaluating Morphological Compositional Generalization in Large Language Models
EPFL（瑞士联邦理工学院）；Idiap Research Institute（Idiap研究机构）；Duke University（杜克大学）；Brandeis University（布兰迪斯大学）；LMU Munich（慕尼黑大学）；University of Cambridge（剑桥大学）；Johns Hopkins University（约翰霍普金斯大学）；Università della Svizzera Italiana（瑞士意大利大学）；New York University（纽约大学）
We present VISPROG, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. VISPROG avoids the need for any task-specific training. Instead, it uses the in-context learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing routines, or python functions to produce intermediate outputs that may be consume…
MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness
Hong Kong University of Science and Technology（香港科技大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；UMass Amherst（马萨诸塞大学阿姆赫斯特分校）；William & Mary（威廉与玛丽学院）
Microsoft Research India（微软印度研究院）；University of Oxford（牛津大学）
Compositional generalization is a fundamental trait in humans, allowing us to effortlessly combine known phrases to form novel sentences. Recent works have claimed that standard seq-to-seq models severely lack the ability to compositionally generalize. In this paper, we focus on one-shot primitive generalization as introduced by the popular SCAN benchmark. We demonstrate that modifying the training distribution in simple and intuitive ways enables standard seq-to-seq models to achieve near-perfect generalization performance, thereby showing that their compositional generalization abilities wer…
"Set It Up!": Functional Object Arrangement with Compositional Generative Models
School of Computing, Smart System Institute, National University of Singapore（计算学院、智能系统研究所、新加坡国立大学）；CSAIL, Massachusetts Institute of Technology（计算机科学与人工智能实验室、麻省理工学院）
An Examination of the Compositionality of Large Generative Vision-Language Models
AI Thrust, The Hong University of Science and Technology (Guangzhou)（香港科技大学（广州）AI方向）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
Columbia University（哥伦比亚大学）；New York University（纽约大学）
Recent work in training large language models (LLMs) to follow natural language instructions has opened up exciting opportunities for natural language interface design. Building on the prior success of LLMs in the realm of computer-assisted creativity, we aim to study if LLMs can improve the quality of user-generated content through collaboration. We present CoPoet, a collaborative poetry writing system. In contrast to auto-completing a user's text, CoPoet is controlled by user instructions that specify the attributes of the desired text, such as Write a sentence about `love' or Write a senten…
Compositional Multilingual and Behavioral Attribute Steering
German Research Centre for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Saarland University（萨尔大学）；Centre for European Research in Trusted AI (CERTAIN)（欧洲可信人工智能研究中心）
On Conditional and Compositional Language Model Differentiable Prompting
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔理工学院）；Amazon Alexa（亚马逊Alexa）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Université de Lille（里尔大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；DeepMind（深度思考）；Google Research（谷歌研究院）
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language understanding and reinforcement learning in even simple instruction following scenarios. This joint learning problem is alleviated through expert demonstrations, auxiliary losses, or neural inductive biases. In this paper, we propose an orthogonal approach called Hindsight Generation fo…
KAIST（韩国科学技术院）；LG AI Research（LG人工智能研究院）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）
Recently, Language Models (LMs) instruction-tuned on multiple tasks, also known as multitask-prompted fine-tuning (MT), have shown the capability to generalize to unseen tasks. Previous work has shown that scaling the number of training tasks is the key component in making stronger MT LMs. In this work, we report an unexpected finding that an expert LM fine-tuned on just a single task can outperform an MT LM trained with 300+ different tasks on 11 different unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean accuracy of 3.20% and 1.29%, respectively. This finding casts doub…
Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following
Seoul National University（首尔大学）；Interdisciplinary Program in AI (IPAI), Seoul National University（首尔大学人工智能交叉项目）；AI Institute, Seoul National University（首尔大学人工智能研究院）
Recent work on large language models relies on the intuition that most natural language processing tasks can be described via natural language instructions. Language models trained on these instructions show strong zero-shot performance on several standard datasets. However, these models even though impressive still perform poorly on a wide range of tasks outside of their respective training and evaluation sets. To address this limitation, we argue that a model should be able to keep extending its knowledge and abilities, without forgetting previous skills. In spite of the limited success of C…
FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance
University of Melbourne, Melbourne, Australia；Brain Science Institute, Korea Institute of Science；Department of Computer Science；Engineering, Korea University, Seoul, Republic of Korea；Division of Bio-Medical Science \& Technology, University of Science；Technology KIST School, Seoul, Republic of Korea
Facebook AI Research（Facebook人工智能研究院）；New York University（纽约大学）
We explore using latent natural language instructions as an expressive and compositional representation of complex actions for hierarchical decision making. Rather than directly selecting micro-actions, our agent first generates a latent plan in natural language, which is then executed by a separate model. We introduce a challenging real-time strategy game environment in which the actions of a large number of units must be coordinated across long time scales. We gather a dataset of 76 thousand pairs of instructions and executions from human play, and train instructor and executor models. Exper…
SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence
Shanghai AI Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；University of Sydney（悉尼大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Beihang University（北航大学）
