Vision-and-language Navigation (VLN) task requires an embodied agent to navigate to a remote location following a natural language instruction. Previous methods usually adopt a sequence model (e.g., Transformer and LSTM) as the navigator. In such a paradigm, the sequence model predicts action at each step through a maintained navigation state, which is generally represented as a one-dimensional vector. However, the crucial navigation clues (i.e., object-level environment layout) for embodied navigation task is discarded since the maintained vector is essentially unstructured. In this paper, we propose a novel Structured state-Evolution (SEvol) model to effectively maintain the environment layout clues for VLN. Specifically, we utilise the graph-based feature to represent the navigation state instead of the vector-based state. Accordingly, we devise a Reinforced Layout clues Miner (RLM) to mine and detect the most crucial layout graph for long-term navigation via a customised reinforcement learning strategy. Moreover, the Structured Evolving Module (SEM) is proposed to maintain the structured graph-based state during navigation, where the state is gradually evolved to learn the object-level spatial-temporal relationship. The experiments on the R2R and R4R datasets show that the proposed SEvol model improves VLN models' performance by large margins, e.g., +3% absolute SPL accuracy for NvEM and +8% for EnvDrop on the R2R test set.
Structured Scene Memory for Vision-Language Navigation
Beijing Institute of Technology(北京理工大学);ETH Zurich(苏黎世联邦理工学院);Salesforce Research(Salesforce研究院);Inception Institute of Artificial Intelligence(Inception人工智能研究院)
Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigation
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Department of Computer Science, Emory University(埃默里大学计算机科学系);Department of Computer Science, University of Wisconsin-Madison(威斯康星大学麦迪逊分校计算机科学系);Tsinghua University(清华大学);Innovation and Research and Development Department, BoardWare Information System Company(BoardWare信息系统公司创新与研发部);School of Mathematics, Peking University(北京大学数学学院)
CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigation
Purdue University(普渡大学);Indian Institute of Technology, Kharagpur(印度理工学院卡哈拉格普尔分校);DEVCOM Army Research Lab(DEVCOM陆军研究实验室);National University of Singapore(新加坡国立大学)
Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memory
School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);Jianghuai Advance Technology Center(江淮先进技术中心);Anhui Provincial Key Laboratory of Humanoid Robots(安徽省人形机器人重点实验室);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);University of Macau(澳门大学)
OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representation
The University of Hong Kong(香港大学);Sun Yat-sen University(中山大学);Digital Content Technology Center, Tencent Games(腾讯游戏数字内容技术中心);GuangDong Province Key Laboratory of Information Security Technology(广东省信息安全重点实验室)
ViSA-Enhanced Aerial VLN: A Visual-Spatial Reasoning Enhanced Framework for Aerial Vision-Language Navigation
Tianmushan Laboratory, Beihang University(北航之梦实验室,北京航空航天大学);Hangzhou International Innovation Institute, Beihang University(杭州国际创新研究院,北京航空航天大学);Foshan Graduate School of Innovation, Northeastern University(佛山创新研究生学院,东北大学);School of Aeronautic Science and Engineering, Beihang University(航空科学与工程学院,北京航空航天大学);Faculty of Robot Science and Engineering, Northeastern University(机器人科学与工程学院,东北大学)