In this work, we present an alternative approach to making an agent compositional through the use of a diagnostic classifier. Because of the need for explainable agents in automated decision processes, we attempt to interpret the latent space from an RL agent to identify its current objective in a complex language instruction. Results show that the classification process causes changes in the hidden states which makes them more easily interpretable, but also causes a shift in zero-shot performance to novel instructions. Lastly, we limit the supervisory signal on the classification, and observe a similar but less notable effect.
Learning policies that effectively utilize language instructions in complex, multi-task environments is an important problem in sequential decision-making. While it is possible to condition on the entire language instruction directly, such an approach could suffer from generalization issues. In our work, we propose \emph{Learning Interpretable Skill Abstractions (LISA)}, a hierarchical imitation learning framework that can learn diverse, interpretable primitive behaviors or skills from language-conditioned demonstrations to better generalize to unseen instructions. LISA uses vector quantizatio…
University of Bristol（布里斯托大学）；CortAIx, Thales（泰雷兹公司 CortAIx）
The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Usi…
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Schwartz Reisman Institute for Technology and Society（施瓦茨·赖斯曼技术与社会研究所）
We address the problem of teaching a deep reinforcement learning (RL) agent to follow instructions in multi-task environments. Instructions are expressed in a well-known formal language -- linear temporal logic (LTL) -- and can specify a diversity of complex, temporally extended behaviours, including conditionals and alternative realizations. Our proposed learning approach exploits the compositional syntax and the semantics of LTL, enabling our RL agent to learn task-conditioned policies that generalize to new instructions, not observed during training. To reduce the overhead of learning LTL s…
Hamburg University of Technology（汉堡工业大学）；University of Hamburg（汉堡大学）
This paper focuses on robotic reinforcement learning with sparse rewards for natural language goal representations. An open problem is the sample-inefficiency that stems from the compositionality of natural language, and from the grounding of language in sensory data and actions. We address these issues with three contributions. We first present a mechanism for hindsight instruction replay utilizing expert feedback. Second, we propose a seq2seq model to generate linguistic hindsight instructions. Finally, we present a novel class of language-focused learning tasks. We show that hindsight instr…
Supervised sparse auto-encoders for interpretable and compositional representations
Department of Computer Science, University of Oxford, Oxford, UK（牛津大学计算机科学系）；KAIST AI, Korean Advanced Institute of Science（韩国科学技术高级研究院AI研究所）；Independent researcher（独立研究者）
Standard architectures used in instruction following often struggle on novel compositions of subgoals (e.g. navigating to landmarks or picking up objects) observed during training. We propose a modular architecture for following natural language instructions that describe sequences of diverse subgoals. In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type. A sequence of modules to execute is chosen by learning to segment the instructions and predicting a subgoal type for each segment. When compared to standard, non-modular sequence-to-sequenc…
Facebook AI Research（Facebook人工智能研究院）；New York University（纽约大学）
We explore using latent natural language instructions as an expressive and compositional representation of complex actions for hierarchical decision making. Rather than directly selecting micro-actions, our agent first generates a latent plan in natural language, which is then executed by a separate model. We introduce a challenging real-time strategy game environment in which the actions of a large number of units must be coordinated across long time scales. We gather a dataset of 76 thousand pairs of instructions and executions from human play, and train instructor and executor models. Exper…
Lessons from Neuroscience for AI: How integrating Actions, Compositional Structure and Episodic Memory could enable Safe, Interpretable and Human-Like AI
Towards Compositional Generalization of LLMs via Skill Taxonomy Guided Data Synthesis
State Key Laboratory of Complex & Critical Software Environment, Beihang University（复杂与关键软件环境国家重点实验室，北京航空航天大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Institute of Technology（北京理工大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）；University of Cambridge（剑桥大学）；University of Toronto（多伦多大学）
Compositional Steering of Large Language Models with Steering Tokens
NEC Laboratories Europe（NEC欧洲实验室）；University of Edinburgh（爱丁堡大学）；Center for Artificial Intelligence and Data Science（人工智能与数据科学中心）；University of Würzburg（乌尔姆大学）；CAIR, Ss. Cyril and Methodius University of Skopje（CAIR，斯·西里尔和方法ius大学）
"Set It Up!": Functional Object Arrangement with Compositional Generative Models
School of Computing, Smart System Institute, National University of Singapore（计算学院、智能系统研究所、新加坡国立大学）；CSAIL, Massachusetts Institute of Technology（计算机科学与人工智能实验室、麻省理工学院）
Many aspects of human reasoning, including language, require learning rules from very little data. Humans can do this, often learning systematic rules from very few examples, and combining these rules to form compositional rule-based systems. Current neural architectures, on the other hand, often fail to generalize in a compositional manner, especially when evaluated in ways that vary systematically from training. In this work, we present a neuro-symbolic model which learns entire rule systems from a small set of examples. Instead of directly predicting outputs from inputs, we train our model …
University of Tübingen（蒂宾根大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Amsterdam（阿姆斯特丹大学）
Learning a common representation space between vision and language allows deep networks to relate objects in the image to the corresponding semantic meaning. We present a model that learns a shared Gaussian mixture representation imposing the compositionality of the text onto the visual domain without having explicit location supervision. By combining the spatial transformer with a representation learning approach we learn to split images into separately encoded patches to associate visual and textual representations in an interpretable manner. On variations of MNIST and CIFAR10, our model is …
MAC Net is a compositional attention network designed for Visual Question Answering. We propose a modified MAC net architecture for Natural Language Question Answering. Question Answering typically requires Language Understanding and multi-step Reasoning. MAC net's unique architecture - the separation between memory and control, facilitates data-driven iterative reasoning. This makes it an ideal candidate for solving tasks that involve logical reasoning. Our experiments with 20 bAbI tasks demonstrate the value of MAC net as a data-efficient and interpretable architecture for Natural Language Q…
We present VISPROG, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. VISPROG avoids the need for any task-specific training. Instead, it uses the in-context learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing routines, or python functions to produce intermediate outputs that may be consume…
Interaction and navigation defined by natural language instructions in dynamic environments pose significant challenges for neural agents. This paper focuses on addressing two challenges: handling long sequence of subtasks, and understanding complex human instructions. We propose Episodic Transformer (E.T.), a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions. To improve training, we leverage synthetic instructions as an intermediate representation that decouples understanding the visual appearance of an environment from the var…
Evaluating Morphological Compositional Generalization in Large Language Models
EPFL（瑞士联邦理工学院）；Idiap Research Institute（Idiap研究机构）；Duke University（杜克大学）；Brandeis University（布兰迪斯大学）；LMU Munich（慕尼黑大学）；University of Cambridge（剑桥大学）；Johns Hopkins University（约翰霍普金斯大学）；Università della Svizzera Italiana（瑞士意大利大学）；New York University（纽约大学）
High-level human instructions often correspond to behaviors with multiple implicit steps. In order for robots to be useful in the real world, they must be able to to reason over both motions and intermediate goals implied by human instructions. In this work, we propose a framework for learning representations that convert from a natural-language command to a sequence of intermediate goals for execution on a robot. A key feature of this framework is prospection, training an agent not just to correctly execute the prescribed command, but to predict a horizon of consequences of an action before t…
We propose an approach for mapping natural language instructions and raw observations to continuous control of a quadcopter drone. Our model predicts interpretable position-visitation distributions indicating where the agent should go during execution and where it should stop, and uses the predicted distributions to select the actions to execute. This two-step model decomposition allows for simple and efficient training using a combination of supervised learning and imitation learning. We evaluate our approach with a realistic drone simulator, and demonstrate absolute task-completion accuracy …
Paul G. Allen School of Computer Sci. & Eng., Univ. of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Allen Institute for AI（艾伦人工智能研究所）；NVIDIA（英伟达公司）
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with non-reversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like "Rinse off a mug and place it in the coffee maker." and low-level l…
