We explore the use of language as a perceptual representation for vision-and-language navigation (VLN), with a focus on low-data settings. Our approach uses off-the-shelf vision systems for image captioning and object detection to convert an agent's egocentric panoramic view at each time step into natural language descriptions. We then finetune a pretrained language model to select an action, based on the current view and the trajectory history, that would best fulfill the navigation instructions. In contrast to the standard setup which adapts a pretrained language model to work directly with continuous visual features from pretrained vision models, our approach instead uses (discrete) language as the perceptual representation. We explore several use cases of our language-based navigation (LangNav) approach on the R2R VLN benchmark: generating synthetic trajectories from a prompted language model (GPT-4) with which to finetune a smaller language model; domain transfer where we transfer a policy learned on one simulated environment (ALFRED) to another (more realistic) environment (R2R); and combining both vision- and language-based representations for VLN. Our approach is found to improve upon baselines that rely on visual features in settings where only a few expert trajectories (10-100) are available, demonstrating the potential of language as a perceptual representation for navigation.
MLFM: Multi-Layered Feature Maps for Richer Language Understanding in Zero-Shot Semantic Navigation
Simon Fraser University(西蒙弗雷泽大学);University of Padova(帕多瓦大学);Fondazione Bruno Kessler (FBK)(布鲁诺·克塞勒基金会);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔人工智能研究所)
Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments
Australian Institute for Machine Learning, The University of Adelaide(阿德莱德大学澳大利亚机器学习研究所);The University of Queensland(昆士兰大学);Zhejiang University of Technology(浙江工业大学)
Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);The Institute of Artificial Intelligence and Robotics(人工智能与机器人研究所);School of Information Science and Technology(信息科学与技术学院);Zhongguancun Academy(中关村学院)
Uncertainty-Aware Gaussian Map for Vision-Language Navigation
The State Key Lab of Brain-Machine Intelligence(脑机智能国家重点实验室);Department of Foundation model, 2012 Labs, Huawei(基础模型部门,2012实验室,华为);Noah’s Ark Lab, 2012 Labs, Huawei(诺亚方舟实验室,2012实验室,华为);School of Software Technology, Zhejiang University(浙江大学软件学院)
Spatial-VLN: Zero-Shot Vision-and-Language Navigation With Explicit Spatial Perception and Exploration
Robotics and Control Laboratory, School of Advanced Manufacturing and Robotics, and the State Key Laboratory of Turbulence and Complex Systems, Peking University(机器人与控制实验室、先进制造与机器人学院,以及湍流与复杂系统国家重点实验室,北京大学);Defense Innovation Institute, Academy of Military Sciences, Beijing(国防科技创新研究院,军事科学院,北京);Tianjin Artificial Intelligence Innovation Center, Tianjin(天津人工智能创新中心,天津);Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)(计算与智能学院,哈尔滨工业大学(深圳))
Emergence of Maps in the Memories of Blind Navigation Agents
Georgia Institute of Technology(佐治亚理工学院);FAIR, Meta AI(元人工智能基础人工智能研究院);Simon Fraser University(西蒙弗雷泽大学);Google Research Atlanta(谷歌亚特兰大研究院);Oregon State University(俄勒冈州立大学)
Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);New York University(纽约大学);Swiss Federal Institute of Technology Lausanne (EPFL)(洛桑联邦理工学院)