We present Success weighted by Completion Time (SCT), a new metric for evaluating navigation performance for mobile robots. Several related works on navigation have used Success weighted by Path Length (SPL) as the primary method of evaluating the path an agent makes to a goal location, but SPL is limited in its ability to properly evaluate agents with complex dynamics. In contrast, SCT explicitly takes the agent's dynamics model into consideration, and aims to accurately capture how well the agent has approximated the fastest navigation behavior afforded by its dynamics. While several embodied navigation works use point-turn dynamics, we focus on unicycle-cart dynamics for our agent, which better exemplifies the dynamics model of popular mobile robotics platforms (e.g., LoCoBot, TurtleBot, Fetch, etc.). We also present RRT*-Unicycle, an algorithm for unicycle dynamics that estimates the fastest collision-free path and completion time from a starting pose to a goal location in an environment containing obstacles. We experiment with deep reinforcement learning and reward shaping to train and compare the navigation performance of agents with different dynamics models. In evaluating these agents, we show that in contrast to SPL, SCT is able to capture the advantages in navigation speed a unicycle model has over a simpler point-turn model of dynamics. Lastly, we show that we can successfully deploy our trained models and algorithms outside of simulation in the real world. We embody our agents in an real robot to navigate an apartment, and show that they can generalize in a zero-shot manner.
Tufts University（塔夫茨大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）；Carnegie Mellon University（卡内基梅隆大学）
Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about obje…
CausalNav: A Long-term Embodied Navigation System for Autonomous Mobile Robots in Dynamic Outdoor Scenarios
Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University（人工智能与机器人中心，深圳国际研究生院，清华大学）；Peng Cheng Laboratory（鹏城实验室）
Toshiba Europe Limited（东芝欧洲有限公司）；University of Cambridge（剑桥大学）
We present NavACL, a method of automatic curriculum learning tailored to the navigation task. NavACL is simple to train and efficiently selects relevant tasks using geometric features. In our experiments, deep reinforcement learning agents trained using NavACL significantly outperform state-of-the-art agents trained with uniform sampling -- the current standard. Furthermore, our agents can navigate through unknown cluttered indoor environments to semantically-specified targets using only RGB images. Obstacle-avoiding policies and frozen feature networks support transfer to unseen real-world en…
University of California, Riverside（加州大学河滨分校）；Mitsubishi Electric Research Labs（三菱电机研究实验室）
Recent years have seen embodied visual navigation advance in two distinct directions: (i) in equipping the AI agent to follow natural language instructions, and (ii) in making the navigable world multimodal, e.g., audio-visual navigation. However, the real world is not only multimodal, but also often complex, and thus in spite of these advances, agents still need to understand the uncertainty in their actions and seek instructions to navigate. To this end, we present AVLEN~ -- an interactive agent for Audio-Visual-Language Embodied Navigation. Similar to audio-visual navigation tasks, the goal…
Facebook AI Research (FAIR)（脸书人工智能研究院）；Facebook Reality Labs (FRL)（脸书现实实验室）
Recent advances in deep reinforcement learning require a large amount of training data and generally result in representations that are often over specialized to the target task. In this work, we present a methodology to study the underlying potential causes for this specialization. We use the recently proposed projection weighted Canonical Correlation Analysis (PWCCA) to measure the similarity of visual representations learned in the same environment by performing different tasks. We then leverage our proposed methodology to examine the task dependence of visual representations learned on rel…
Georgia Institute of Technology（佐治亚理工学院）；Meta AI；Google Atlanta（谷歌亚特兰大）
We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). VER learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL). VER leads to significant and consistent speed-ups across a bro…
NaviTrace: Evaluating Embodied Navigation of Vision-Language Models
Robotic Systems Lab, ETH Zurich（苏黎世联邦理工学院机器人系统实验室）；Intuitive Robots Lab, KIT（卡尔斯鲁厄理工学院直观机器人实验室）；FZI Research Center for Information Technology（弗劳恩霍夫信息技术研究中心）
NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
RIKEN Center for AIP（理化学研究所革新智能综合研究中心）；Tohoku University（东北大学）
Visual exploration is a task that seeks to visit all the navigable areas of an environment as quickly as possible. The existing methods employ deep reinforcement learning (RL) as the standard tool for the task. However, they tend to be vulnerable to statistical shifts between the training and test data, resulting in poor generalization over novel environments that are out-of-distribution (OOD) from the training data. In this paper, we attempt to improve the generalization ability by utilizing the inductive biases available for the task. Employing the active neural SLAM (ANS) that learns explor…
Facebook AI Research（Facebook人工智能研究）；Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual represe…
Georgia Institute of Technology（佐治亚理工学院）；Oregon State University（俄勒冈州立大学）；Google（谷歌）；Facebook AI Research (FAIR)（Facebook人工智能研究院（FAIR））
Following a navigation instruction such as 'Walk down the stairs and stop at the brown sofa' requires embodied AI agents to ground scene elements referenced via language (e.g. 'stairs') to visual content in the environment (pixels corresponding to 'stairs'). We ask the following question -- can we leverage abundant 'disembodied' web-scraped vision-and-language corpora (e.g. Conceptual Captions) to learn visual groundings (what do 'stairs' look like?) that improve performance on a relatively data-starved embodied perception task (Vision-and-Language Navigation)? Specifically, we develop VLN-BER…
State Key Laboratory of Robotics and Intelligent Systems（机器人与智能系统国家重点实验室）；Shenyang Institute of Automation（沈阳自动化研究所）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
PM-Nav: Priori-Map Guided Embodied Navigation in Functional Buildings
Faculty of Robot Science and Engineering at Northeastern University（东北大学机器人科学与工程学院）；Foshan Graduate School of Innovation at Northeastern University（东北大学佛山创新研究生学院）；School of Aeronautic Science and Engineering at Beihang University（北航航空科学与工程学院）
A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI
Department of Computer Science, City University of Hong Kong（城市大学计算机科学系）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；Department of Informatics, Universität Hamburg（汉堡大学信息学院）
Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigation
Southwestern University of Finance（西南财经大学）；Department of Informatics,Universitat Hamburg, Hamburg, Germany（汉堡大学信息学院）；University of Electronic Science（电子科技大学）
Monash University（莫纳什大学）；University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-sen University（中山大学）；RMIT University（皇家墨尔本理工大学）
"Embodied visual navigation" problem requires an agent to navigate in a 3D environment mainly rely on its first-person observation. This problem has attracted rising attention in recent years due to its wide application in autonomous driving, vacuum cleaner, and rescue robot. A navigation agent is supposed to have various intelligent skills, such as visual perceiving, mapping, planning, exploring and reasoning, etc. Building such an agent that observes, thinks, and acts is a key to real intelligence. The remarkable learning ability of deep learning methods empowered the agents to accomplish em…
Towards Path-Creative Navigation: Robot Navigation through Embodied Interaction
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；College of Information Science and Technology, Eastern Institute of Technology（东方理工大学信息科学与技术学院）；School of Mechanical and Power Engineering, Zhengzhou university（郑州大学机械与动力工程学院）；University of Waterloo（滑铁卢大学）
Sensing, Social, and Motion Intelligence in Embodied Navigation: A Comprehensive Survey
Shanghai Jiao Tong University（上海交通大学）；Harbin Engineering University（哈尔滨工程大学）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Zhejiang University（浙江大学）
Georgia Tech（佐治亚理工学院）；Allen Institute of AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）
As an attempt towards assessing the robustness of embodied navigation agents, we propose RobustNav, a framework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual - affecting RGB inputs - and dynamics - affecting transition dynamics - corruptions. Most recent efforts in visual navigation have typically focused on generalizing to novel target environments with similar appearance and dynamics characteristics. With RobustNav, we find that some standard embodied navigation agents significantly underperform (or fail) in the presence of visual or dynam…
University of Padova（帕多瓦大学）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会（FBK））；Simon Fraser University（西蒙弗雷泽大学）
Learning how to navigate among humans in an occluded and spatially constrained indoor environment, is a key ability required to embodied agent to be integrated into our society. In this paper, we propose an end-to-end architecture that exploits Proximity-Aware Tasks (referred as to Risk and Proximity Compass) to inject into a reinforcement learning navigation policy the ability to infer common-sense social behaviors. To this end, our tasks exploit the notion of immediate and future dangers of collision. Furthermore, we propose an evaluation protocol specifically designed for the Social Navigat…
Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；UAS Esslingen（埃森嫩大学）；TU Wien（维也纳技术大学）；Dr. Ing. h.c. F. Porsche AG（保时捷股份有限公司）；University of Michigan（密歇根大学）；Voxel51 Inc.（Voxel51公司）
FloNa: Floor Plan Guided Embodied Visual Navigation
Beijing Institute of Technology（北京理工大学）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）；Beijing Racobit Electronic Information Technology Co., Ltd.（北京睿科比特电子信息技术有限公司）
Meta AI Research（Meta人工智能研究院）；Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Realistic long-horizon tasks like image-goal navigation involve exploratory and exploitative phases. Assigned with an image of the goal, an embodied agent must explore to discover the goal, i.e., search efficiently using learned priors. Once the goal is discovered, the agent must accurately calibrate the last-mile of navigation to the goal. As with any robust system, switches between exploratory goal discovery and exploitative last-mile navigation enable better recovery from errors. Following these intuitive guide rails, we propose SLING to improve the performance of existing image-goal naviga…
Human-robot collaboration is an essential research topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human languages facilitate communication by instructions in natural languages. Nevertheless, a plethora of other forms of communication is left unexplored. In fact, human communication originated in gestures and oftentimes is delivered through multimodal cues, e.g…
Australian National University（澳大利亚国立大学）；Princeton University（普林斯顿大学）；Carnegie Mellon University（卡内基梅隆大学）；Intel Labs（英特尔实验室）；UC Berkeley（加州大学伯克利分校）；George Mason University（乔治梅森大学）；Facebook（脸书）；Allen Institute for AI（艾伦人工智能研究所）
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This creative output has produced a plethora of sometimes incompatible task definitions and evaluation protocols. To coordinate ongoing and future research in this area, we have convened a working group to study empirical methodology in navigation research. The present document summarizes the consensus recommendations of this working group. We discuss different problem statements and the role of generalization, …
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Trustworthy Autonomous Systems Laboratory at the University of California, Riverside（加州大学河滨分校可信自主系统实验室）；University of Michigan（密歇根大学）；Workday；University of Southern California（南加州大学）；Texas A&M University（德克萨斯A&M大学）；Lehigh University（莱斯大学）
University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学）
The research field of Embodied AI has witnessed substantial progress in visual navigation and exploration thanks to powerful simulating platforms and the availability of 3D data of indoor and photorealistic environments. These two factors have opened the doors to a new generation of intelligent agents capable of achieving nearly perfect PointGoal Navigation. However, such architectures are commonly trained with millions, if not billions, of frames and tested in simulation. Together with great enthusiasm, these results yield a question: how many researchers will effectively benefit from these a…
