Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents. The 'interactive instruction following' task attempts to make progress towards building agents that jointly navigate, interact, and reason in the environment at every step. To address the multifaceted problem, we propose a model that factorizes the task into interactive perception and action policy streams with enhanced components and name it as MOCA, a Modular Object-Centric Approach. We empirically validate that MOCA outperforms prior arts by significant margins on the ALFRED benchmark with improved generalization.
Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multi-modal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to…
In cooperation, the workers must know how co-workers behave. However, an agent's policy, which is embedded in a statistical machine learning model, is hard to understand, and requires much time and knowledge to comprehend. Therefore, it is difficult for people to predict the behavior of machine learning robots, which makes Human Robot Cooperation challenging. In this paper, we propose Instruction-based Behavior Explanation (IBE), a method to explain an autonomous agent's future behavior. In IBE, an agent can autonomously acquire the expressions to explain its own behavior by reusing the instru…
IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents
Massachusetts Institute of Technology（麻省理工学院）；University of Waterloo（滑铁卢大学）；Meta AI；AIRI（人工智能研究院）；École Polytechnique de Montréal（蒙特利尔理工学院）；Microsoft Research（微软研究院）
Microsoft Research（微软研究院）；Alexa AI；University of Amsterdam（阿姆斯特丹大学）；MIPT（莫斯科物理技术学院）；AIRI；Facebook AI（脸书人工智能研究院）
Human intelligence has the remarkable ability to adapt to new tasks and environments quickly. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research in this direction, we propose IGLU: Interactive Grounded Language Understanding in a Collaborative Environment. The primary goal of the competition is to approach the problem of how to build interactive agents that learn to solve a task while provided with grounded natural language instructio…
Behavioral skills or policies for autonomous agents are conventionally learned from reward functions, via reinforcement learning, or from demonstrations, via imitation learning. However, both modes of task specification have their disadvantages: reward functions require manual engineering, while demonstrations require a human expert to be able to actually perform the task in order to generate the demonstration. Instruction following from natural language instructions provides an appealing alternative: in the same way that we can specify goals to other humans simply by speaking or writing, we w…
While traditional methods for instruction-following typically assume prior linguistic and perceptual knowledge, many recent works in reinforcement learning (RL) have proposed learning policies end-to-end, typically by training neural networks to map joint representations of observations and instructions directly to actions. In this work, we present a novel framework for learning to perform temporally extended tasks using spatial reasoning in the RL framework, by sequentially imagining visual goals and choosing appropriate actions to fulfill imagined goals. Our framework operates on raw pixel i…
University of Amsterdam（阿姆斯特丹大学）；Facebook A.I. Research（脸书人工智能研究院）；Universitat Pompeu Fabra（庞培法布拉大学）
To cooperate with humans effectively, virtual agents need to be able to understand and execute language instructions. A typical setup to achieve this is with a scripted teacher which guides a virtual agent using language instructions. However, such setup has clear limitations in scalability and, more importantly, it is not interactive. Here, we introduce an autonomous agent that uses discrete communication to interactively guide other agents to navigate and act on a simulated environment. The developed communication protocol is trainable, emergent and requires no additional supervision. The em…
Learning novel tasks is a complex cognitive activity requiring the learner to acquire diverse declarative and procedural knowledge. Prior ACT-R models of acquiring task knowledge from instruction focused on learning procedural knowledge from declarative instructions encoded in semantic memory. In this paper, we identify the requirements for designing compu- tational models that learn task knowledge from situated task- oriented interactions with an expert and then describe and evaluate a model of learning from situated interactive instruc- tion that is implemented in the Soar cognitive architec…
Federation University（联邦大学）；Deakin University（迪肯大学）；Universidad Central de Chile（智利中央大学）
Interactive reinforcement learning proposes the use of externally-sourced information in order to speed up the learning process. When interacting with a learner agent, humans may provide either evaluative or informative advice. Prior research has focused on the effect of human-sourced advice by including real-time feedback on the interactive reinforcement learning process, specifically aiming to improve the learning speed of the agent, while minimising the time demands on the human. This work focuses on answering which of two approaches, evaluative or informative, is the preferred instructiona…
One significant simplification in most previous work on robot learning is the closed-world assumption where the robot is assumed to know ahead of time a complete set of predicates describing the state of the physical world. However, robots are not likely to have a complete model of the world especially when learning a new task. To address this problem, this extended abstract gives a brief introduction to our on-going work that aims to enable the robot to acquire new state representations through language communication with humans.
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）；Kyoto University（京都大学）
An interactive instruction following task has been proposed as a benchmark for learning to map natural language instructions and first-person vision into sequences of actions to interact with objects in 3D environments. We found that an existing end-to-end neural model for this task tends to fail to interact with objects of unseen attributes and follow various instructions. We assume that this problem is caused by the high sensitivity of neural feature extraction to small changes in vision and language inputs. To mitigate this problem, we propose a neuro-symbolic approach that utilizes high-le…
Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions
Microsoft Research（微软研究院）；University of Waterloo（滑铁卢大学）；École Polytechnique de Montréal（蒙特利尔综合理工学院）；Universidad Nacional de Córdoba（科尔多瓦国立大学）；Meta AI；DeepMind（深度思维）
Recently, text-guided image manipulation has received increasing attention in the research field of multimedia processing and computer vision due to its high flexibility and controllability. Its goal is to semantically manipulate parts of an input reference image according to the text descriptions. However, most of the existing works have the following problems: (1) text-irrelevant content cannot always be maintained but randomly changed, (2) the performance of image manipulation still needs to be further improved, (3) only can manipulate descriptive attributes. To solve these problems, we pro…
PaniniQA: Enhancing Patient Education Through Interactive Question Answering
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Emory University（埃默里大学）；Northeastern University（东北大学）；UMass Chan Medical School（马萨诸塞大学陈医学院）；University of Minnesota（明尼苏达大学）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model
Waseda University（早稻田大学）；Fujitsu Limited（富士通有限公司）；Waseda Research Institute for Science and Engineering（早稻田大学科学与工程研究院）；National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）
This paper presents an approach to learning from situated, interactive tutorial instruction within an ongoing agent. Tutorial instruction is a flexible (and thus powerful) paradigm for teaching tasks because it allows an instructor to communicate whatever types of knowledge an agent might need in whatever situations might arise. To support this flexibility, however, the agent must be able to learn multiple kinds of knowledge from a broad range of instructional interactions. Our approach, called situated explanation, achieves such learning through a combination of analytic and inductive techniq…
University of Washington（华盛顿大学）；NVIDIA（英伟达）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Allen Institute for AI（艾伦人工智能研究所）
Widespread adoption of self-driving cars will depend not only on their safety but largely on their ability to interact with human users. Just like human drivers, self-driving cars will be expected to understand and safely follow natural-language directions that suddenly alter the pre-planned route according to user's preference or in presence of ambiguities, particularly in locations with poor or outdated map coverage. To this end, we propose a language-grounded driving agent implementing a hierarchical policy using recurrent layers and gated attention. The hierarchical approach enables us to …
Microsoft Research（微软研究院）；University of Amsterdam（阿姆斯特丹大学）；Alexa AI（Alexa人工智能）；MIPT（莫斯科物理技术学院）；AIRI（人工智能研究院）；Facebook AI（脸书人工智能）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Kalashnikov Izhevsk State Technical University（卡拉什尼科夫伊热夫斯克国立技术大学）；Universiteit Leiden（莱顿大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）
Human intelligence has the remarkable ability to quickly adapt to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research in this direction, we propose \emph{IGLU: Interactive Grounded Language Understanding in a Collaborative Environment}. The primary goal of the competition is to approach the problem of how to build interactive agents that learn to solve a task while provided with grounded natural language ins…
AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning
School of Computer Science, Hangzhou Dianzi University（杭州电子科技大学计算机学院）；State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Fullong Inc.（丰隆公司）；NingBo Port Group（宁波港集团）
Paul G. Allen School of Computer Sci. & Eng., Univ. of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Allen Institute for AI（艾伦人工智能研究所）；NVIDIA（英伟达公司）
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with non-reversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like "Rinse off a mug and place it in the coffee maker." and low-level l…
Natural language instruction following tasks serve as a valuable test-bed for grounded language and robotics research. However, data collection for these tasks is expensive and end-to-end approaches suffer from data inefficiency. We propose the structuring of language, acting, and visual tasks into separate modules that can be trained independently. Using a Language, Action, and Vision (LAV) framework removes the dependence of action and vision modules on instruction following datasets, making them more efficient to train. We also present a preliminary evaluation of LAV on the ALFRED task for …
InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
Shenzhen Institute of Advanced Technology Chinese Academy of Sciences（深圳先进技术研究院中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Dalian University of Technology（大连理工大学）；UNSW Sydney（悉尼大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Microsoft Research（微软研究院）；University of Waterloo（滑铁卢大学）；Meta Research（Meta研究院）；Moscow Institute of Physics Technology（莫斯科物理技术学院）；Artificial Intelligence Research Institute（人工智能研究院）
Human intelligence can remarkably adapt quickly to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research which can enable similar capabilities in machines, we made the following contributions (1) formalized the collaborative embodied agent using natural language task; (2) developed a tool for extensive and scalable data collection; and (3) collected the first dataset for interactive grounded language understan…
LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation
NetEase, Inc.（网易公司）；Beihang University（北京航空航天大学）；Tsinghua University（清华大学）；City University of Hong Kong（香港城市大学）；Independent Researcher & Technical Artists（独立研究者及技术艺术家）
