Embodied instruction following is a challenging problem requiring an agent to infer a sequence of primitive actions to achieve a goal environment state from complex language and visual inputs. Action Learning From Realistic Environments and Directives (ALFRED) is a recently proposed benchmark for this problem consisting of step-by-step natural language instructions to achieve subgoals which compose to an ultimate high-level goal. Key challenges for this task include localizing target locations and navigating to them through visual inputs, and grounding language instructions to visual appearance of objects. To address these challenges, in this study, we augment the agent's field of view during navigation subgoals with multiple viewing angles, and train the agent to predict its relative spatial relation to the target location at each timestep. We also improve language grounding by introducing a pre-trained object detection module to the model pipeline. Empirical studies show that our approach exceeds the baseline model performance.
Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following
Seoul National University（首尔大学）；Interdisciplinary Program in AI (IPAI), Seoul National University（首尔大学人工智能交叉项目）；AI Institute, Seoul National University（首尔大学人工智能研究院）
NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）
AIRI；MIPT（莫斯科物理技术学院）；Microsoft Research（微软研究院）；University of Waterloo（滑铁卢大学）
The adoption of pre-trained language models to generate action plans for embodied agents is a promising research strategy. However, execution of instructions in real or simulated environments requires verification of the feasibility of actions as well as their relevance to the completion of a goal. We propose a new method that combines a language model and reinforcement learning for the task of building objects in a Minecraft-like environment according to the natural language instructions. Our method first generates a set of consistently achievable sub-goals from the instructions and then comp…
RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation
MBZUAI（默罕默德·本·扎耶德人工智能大学）；Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）；AAII, UTS（悉尼科技大学人工智能与创新研究所）；University of Science and Technology of China（中国科学技术大学）
RoboGPT: an intelligent agent of making embodied long-term decisions for daily instruction tasks
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；Center on Frontiers of Computing Studies, Peking University（北京大学计算科学前沿中心）
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Trustworthy Autonomous Systems Laboratory at the University of California, Riverside（加州大学河滨分校可信自主系统实验室）；University of Michigan（密歇根大学）；Workday；University of Southern California（南加州大学）；Texas A&M University（德克萨斯A&M大学）；Lehigh University（莱斯大学）
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Institute of Trustworthy Embodied AI, Fudan University（复旦大学可信具身人工智能研究院）
Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（信息多媒体国家重点实验室，计算机学院，北京大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute for Brain and Intelligence, Fudan University（脑与智能研究院，复旦大学）；University of Science and Technology Beijing（北京科技大学）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）
A Multimodal Depth-Aware Method For Embodied Reference Understanding
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；KIT Campus Transfer GmbH (KCT)（KIT校园转移有限公司）；Istanbul Technical University（伊斯坦布尔技术大学）；Carnegie Mellon University（卡内基梅隆大学）
Diagnose, Localize, Align: A Full-Stack Framework for Reliable LLM Multi-Agent Systems under Instruction Conflicts
University of California, Los Angeles（加州大学洛杉矶分校）；Sea AI Lab（Sea AI 实验室）；Stanford University（斯坦福大学）；University of Oxford（牛津大学）；Yale University（耶鲁大学）；NTU（南洋理工大学）；NUS（新加坡国立大学）；Boston University（波士顿大学）
GuardianBench: A Same-Scene Instruction-Contrastive Benchmark for Latent Contextual Risk in Embodied AI
Shandong University（山东大学）；National University of Singapore（新加坡国立大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Xiaomi Corporation（小米公司）
VLP: Vision-Language Preference Learning for Embodied Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院）；King’s College London（伦敦国王学院）
UCLA（加州大学洛杉矶分校）；Amazon Alexa AI（亚马逊Alexa人工智能）；USC Viterbi School of Engineering（南加州大学维特比工程学院）
Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects typically allow one-way communication: the human user gives a natural language command to the agent, and the agent can only follow the command passively. We present DialFRED, a dialogue-enabled embodied instruction following benchmark based on the ALFRED benchmark. DialFRED allows an agent to actively ask questions to the human user; the additional information in the user's response is used by the agent to better complete its task. We release a human-annotated dataset with 53K task-relev…
Embodied Instruction Following in Unknown Environments
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications（北京邮电大学智能工程与自动化学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；Department of Automation, Tsinghua University（清华大学自动化学院）
Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions
Microsoft Research（微软研究院）；University of Waterloo（滑铁卢大学）；École Polytechnique de Montréal（蒙特利尔综合理工学院）；Universidad Nacional de Córdoba（科尔多瓦国立大学）；Meta AI；DeepMind（深度思维）
Renmin University of China（中国人民大学）；Samsung Research China Beijing（三星中国研究院（北京））
People always desire an embodied agent that can perform a task by understanding language instruction. Moreover, they also want to monitor and expect agents to understand commands the way they expected. But, how to build such an embodied agent is still unclear. Recently, people can explore this problem with the Vision-and-Language Interaction benchmark ALFRED, which requires an agent to perform complicated daily household tasks following natural language instructions in unseen scenes. In this paper, we propose LEBP -- Language Expectation and Binding Policy Module to tackle the ALFRED. The LEBP…
Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation
The Hong Kong Polytechnic University（香港理工大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Great Bay University（大湾区大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Limited Linguistic Diversity in Embodied AI Datasets
Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Institute of Computer Science, University of Tartu（塔尔图大学计算机科学学院）；Department of Mechanical Engineering, The University of Texas at Austin（德克萨斯大学奥斯汀分校机械工程系）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）
Recent years have seen an increasing amount of work on embodied AI agents that can perform tasks by following human language instructions. However, most of these agents are reactive, meaning that they simply learn and imitate behaviors encountered in the training data. These reactive agents are insufficient for long-horizon complex tasks. To address this limitation, we propose a neuro-symbolic deliberative agent that, while following language instructions, proactively applies reasoning and planning based on its neural and symbolic representations acquired from past experience (e.g., natural la…
