Deep Reinforcement Learning (DRL) has shown great potential in enabling robots to find certain objects (e.g., `find a fridge') in environments like homes or schools. This task is known as Object-Goal Navigation (ObjectNav). DRL methods are predominantly trained and evaluated using environment simulators. Although DRL has shown impressive results, the simulators may be biased or limited. This creates a risk of shortcut learning, i.e., learning a policy tailored to specific visual details of training environments. We aim to deepen our understanding of shortcut learning in ObjectNav, its implications and propose a solution. We design an experiment for inserting a shortcut bias in the appearance of training environments. As a proof-of-concept, we associate room types to specific wall colors (e.g., bedrooms with green walls), and observe poor generalization of a state-of-the-art (SOTA) ObjectNav method to environments where this is not the case (e.g., bedrooms with blue walls). We find that shortcut learning is the root cause: the agent learns to navigate to target objects, by simply searching for the associated wall color of the target object's room. To solve this, we propose Language-Based (L-B) augmentation. Our key insight is that we can leverage the multimodal feature space of a Vision-Language Model (VLM) to augment visual representations directly at the feature-level, requiring no changes to the simulator, and only an addition of one layer to the model. Where the SOTA ObjectNav method's success rate drops 69%, our proposal has only a drop of 23%.
Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；Hunan Artificial Intelligence and Robotics Institute Company Ltd.（湖南人工智能与机器人研究院有限公司）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Peng Cheng Laboratory（鹏城实验室）
GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Robbyant；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；The Hong Kong University of Science and Technology（香港科技大学）
Look Beyond Bias with Entropic Adversarial Data Augmentation
Univ Lyon（里昂大学）；Ecole Centrale de Lyon（里昂中央理工学院）；CNRS（法国国家科学研究中心）；INSA Lyon（里昂国立应用科学学院）；Univ Claude Bernard Lyon 1（里昂第一大学）；Univ Louis Lumière Lyon 2（里昂第二大学）；LIRIS（里昂信息与图像研究所）；Thales LAS France（法国泰雷兹激光系统公司）
Emergence of Maps in the Memories of Blind Navigation Agents
Georgia Institute of Technology（佐治亚理工学院）；FAIR, Meta AI（元人工智能基础人工智能研究院）；Simon Fraser University（西蒙弗雷泽大学）；Google Research Atlanta（谷歌亚特兰大研究院）；Oregon State University（俄勒冈州立大学）
DyNaVLM: Zero-Shot Vision-Language Navigation System with Dynamic Viewpoints and Self-Refining Graph Memory
SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University（上海交通大学巴黎精英技术研究所）；Department of Automation, Shanghai Jiao Tong University（上海交通大学自动化系）；MoE Key Lab of Artificial Intelligence and AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能与AI研究所）
