In language-guided visual navigation, agents locate target objects in unseen environments using natural language instructions. For reliable navigation in unfamiliar scenes, agents should possess strong perception, planning, and prediction capabilities. Additionally, when agents revisit previously explored areas during long-term navigation, they may retain irrelevant and redundant historical perceptions, leading to suboptimal results. In this work, we propose RoboTron-Nav, a unified framework that integrates perception, planning, and prediction capabilities through multitask collaborations on navigation and embodied question answering tasks, thereby enhancing navigation performances. Furthermore, RoboTron-Nav employs an adaptive 3D-aware history sampling strategy to effectively and efficiently utilize historical observations. By leveraging large language model, RoboTron-Nav comprehends diverse commands and complex visual scenes, resulting in appropriate navigation actions. RoboTron-Nav achieves an 81.1% success rate in object goal navigation on the $\mathrm{CHORES}$-$\mathbb{S}$ benchmark, setting a new state-of-the-art performance. Project page: https://yvfengzhong.github.io/RoboTron-Nav
Towards Path-Creative Navigation: Robot Navigation through Embodied Interaction
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；College of Information Science and Technology, Eastern Institute of Technology（东方理工大学信息科学与技术学院）；School of Mechanical and Power Engineering, Zhengzhou university（郑州大学机械与动力工程学院）；University of Waterloo（滑铁卢大学）
Monash University（莫纳什大学）；University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-sen University（中山大学）；RMIT University（皇家墨尔本理工大学）
"Embodied visual navigation" problem requires an agent to navigate in a 3D environment mainly rely on its first-person observation. This problem has attracted rising attention in recent years due to its wide application in autonomous driving, vacuum cleaner, and rescue robot. A navigation agent is supposed to have various intelligent skills, such as visual perceiving, mapping, planning, exploring and reasoning, etc. Building such an agent that observes, thinks, and acts is a key to real intelligence. The remarkable learning ability of deep learning methods empowered the agents to accomplish em…
Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；UAS Esslingen（埃森嫩大学）；TU Wien（维也纳技术大学）；Dr. Ing. h.c. F. Porsche AG（保时捷股份有限公司）；University of Michigan（密歇根大学）；Voxel51 Inc.（Voxel51公司）
Georgia Tech（佐治亚理工学院）；Allen Institute of AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）
As an attempt towards assessing the robustness of embodied navigation agents, we propose RobustNav, a framework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual - affecting RGB inputs - and dynamics - affecting transition dynamics - corruptions. Most recent efforts in visual navigation have typically focused on generalizing to novel target environments with similar appearance and dynamics characteristics. With RobustNav, we find that some standard embodied navigation agents significantly underperform (or fail) in the presence of visual or dynam…
A Deployable Embodied Vision-Language Navigation System with Hierarchical Cognition and Context-Aware Exploration
Center for Advanced Robotics Technology Innovation (CARTIN), School of Electrical and Electronic Engineering, Nanyang Technological University（先进机器人技术与创新中心（CARTIN）、电子工程学院、南洋理工大学）；Spatial AI & Robotics Lab, Department of Computer Science and Engineering（空间人工智能与机器人实验室、计算机科学与工程系）
Toshiba Europe Limited（东芝欧洲有限公司）；University of Cambridge（剑桥大学）
We present NavACL, a method of automatic curriculum learning tailored to the navigation task. NavACL is simple to train and efficiently selects relevant tasks using geometric features. In our experiments, deep reinforcement learning agents trained using NavACL significantly outperform state-of-the-art agents trained with uniform sampling -- the current standard. Furthermore, our agents can navigate through unknown cluttered indoor environments to semantically-specified targets using only RGB images. Obstacle-avoiding policies and frozen feature networks support transfer to unseen real-world en…
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Trustworthy Autonomous Systems Laboratory at the University of California, Riverside（加州大学河滨分校可信自主系统实验室）；University of Michigan（密歇根大学）；Workday；University of Southern California（南加州大学）；Texas A&M University（德克萨斯A&M大学）；Lehigh University（莱斯大学）
University of British Columbia（不列颠哥伦比亚大学）；National University of Singapore（新加坡国立大学）；University of Washington（华盛顿大学）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）
This paper presents a framework for training an agent to actively request help in object-goal navigation tasks, with feedback indicating the location of the target object in its field of view. To make the agent more robust in scenarios where a teacher may not always be available, the proposed training curriculum includes a mix of episodes with and without feedback. The results show that this approach improves the agent's performance, even in the absence of feedback.
Tufts University（塔夫茨大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）；Carnegie Mellon University（卡内基梅隆大学）
Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about obje…
VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
Adelaide University（阿德莱德大学）；Responsible AI Research Centre, Australian Institute for Machine Learning（负责任人工智能研究中心、澳大利亚机器学习研究所）；Tongji University（同济大学）；ManyCore；Zhejiang University（浙江大学）；Shanghai AI Lab（上海人工智能实验室）；CSIRO Data61
Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection
LAAS-CNRS, Universite de Toulouse, CNRS（法国国家科学研究中心图卢兹分部LAAS、图卢兹大学、法国国家科学研究中心）；Department of Electrical Engineering and Automation at Aalto University（阿尔托大学电气工程与自动化系）
Meta AI Research（Meta人工智能研究院）；Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Realistic long-horizon tasks like image-goal navigation involve exploratory and exploitative phases. Assigned with an image of the goal, an embodied agent must explore to discover the goal, i.e., search efficiently using learned priors. Once the goal is discovered, the agent must accurately calibrate the last-mile of navigation to the goal. As with any robust system, switches between exploratory goal discovery and exploitative last-mile navigation enable better recovery from errors. Following these intuitive guide rails, we propose SLING to improve the performance of existing image-goal naviga…
FloNa: Floor Plan Guided Embodied Visual Navigation
Beijing Institute of Technology（北京理工大学）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）；Beijing Racobit Electronic Information Technology Co., Ltd.（北京睿科比特电子信息技术有限公司）
OrangeDev s.r.l., Firenze, Italy（OrangeDev s.r.l.，意大利佛罗伦萨）；Cognitive Robotics and Social Sensing LaboratoryICAR-CNR, Palermo, Italy（意大利帕勒莫ICAR-CNR认知机器人与社会感知实验室）
Visual navigation models based on deep learning can learn effective policies when trained on large amounts of visual observations through reinforcement learning. Unfortunately, collecting the required experience in the real world requires the deployment of a robotic platform, which is expensive and time-consuming. To deal with this limitation, several simulation platforms have been proposed in order to train visual navigation policies on virtual environments efficiently. Despite the advantages they offer, simulators present a limited realism in terms of appearance and physical dynamics, leadin…
MA-CoNav: A Master-Slave Multi-Agent Framework with Hierarchical Collaboration and Dual-Level Reflection for Long-Horizon Embodied VLN
Southwestern University of Finance（西南财经大学）；Department of Informatics,Universitat Hamburg, Hamburg, Germany（汉堡大学信息学院）；University of Electronic Science（电子科学大学）
We propose a domain adaptation method, MoDA, which adapts a pretrained embodied agent to a new, noisy environment without ground-truth supervision. Map-based memory provides important contextual information for visual navigation, and exhibits unique spatial structure mainly composed of flat walls and rectangular obstacles. Our adaptation approach encourages the inherent regularities on the estimated maps to guide the agent to overcome the prevalent domain discrepancy in a novel environment. Specifically, we propose an efficient learning curriculum to handle the visual and dynamics corruptions …
Facebook AI Research（Facebook人工智能研究）；Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual represe…
Sensing, Social, and Motion Intelligence in Embodied Navigation: A Comprehensive Survey
Shanghai Jiao Tong University（上海交通大学）；Harbin Engineering University（哈尔滨工程大学）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Zhejiang University（浙江大学）
CausalNav: A Long-term Embodied Navigation System for Autonomous Mobile Robots in Dynamic Outdoor Scenarios
Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University（人工智能与机器人中心，深圳国际研究生院，清华大学）；Peng Cheng Laboratory（鹏城实验室）
Plan in Sandbox, Navigate in Open Worlds: Learning Physics-Grounded Abstracted Experience for Embodied Navigation
School of Computing and Artificial Intelligence, Southwest Jiaotong University, China（计算机与人工智能学院，西南交通大学，中国）；Centre for Frontier AI Research A*STAR, Singapore（前沿人工智能研究A*STAR中心，新加坡）；School of Computer Science, University of Leeds, UK（计算机科学学院，利兹大学，英国）
NaviTrace: Evaluating Embodied Navigation of Vision-Language Models
Robotic Systems Lab, ETH Zurich（苏黎世联邦理工学院机器人系统实验室）；Intuitive Robots Lab, KIT（卡尔斯鲁厄理工学院直观机器人实验室）；FZI Research Center for Information Technology（弗劳恩霍夫信息技术研究中心）
Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；Adelaide University（阿德莱德大学）；Responsible AI Research Centre（负责任人工智能研究中心）；CSIRO Data61（联邦科学与工业研究组织数据61中心）；The University of Queensland（昆士兰大学）
PM-Nav: Priori-Map Guided Embodied Navigation in Functional Buildings
Faculty of Robot Science and Engineering at Northeastern University（东北大学机器人科学与工程学院）；Foshan Graduate School of Innovation at Northeastern University（东北大学佛山创新研究生学院）；School of Aeronautic Science and Engineering at Beihang University（北航航空科学与工程学院）
SpikingNav: Robust Embodied Navigation with Spiking Neural Policies
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学与技术国家研究中心）
State Key Laboratory of Robotics and Intelligent Systems（机器人与智能系统国家重点实验室）；Shenyang Institute of Automation（沈阳自动化研究所）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
