arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.07889cs.CVcs.AIcs.CLcs.RO
收藏

LangNav:语言作为导航的感知表征

LangNav: Language as a Perceptual Representation for Navigation

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);Dartmouth College(达特茅斯学院)

Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, Yoon Kim

AI总结:

本文提出LangNav,将语言描述作为视觉-语言导航的感知表征,在低数据场景下利用合成轨迹、领域迁移及视觉-语言表征融合,超越依赖视觉特征的基线方法。

摘要:

我们探索将语言用作视觉-语言导航(VLN)的感知表征,重点关注低数据场景。我们的方法使用现成的视觉系统进行图像描述和目标检测,将智能体在每个时间步的自我中心全景视图转换为自然语言描述。然后,我们对一个预训练语言模型进行微调,使其根据当前视图和轨迹历史选择最能满足导航指令的动作。与标准设置——即调整预训练语言模型以直接处理来自预训练视觉模型的连续视觉特征——不同,我们的方法使用(离散的)语言作为感知表征。我们在R2R VLN基准上探索了基于语言的导航(LangNav)方法的几种使用场景:利用提示语言模型(GPT-4)生成合成轨迹,再用这些轨迹微调一个较小的语言模型;进行领域迁移,将在一个模拟环境(ALFRED)中学到的策略迁移到另一个(更真实的)环境(R2R);以及将基于视觉和基于语言的表征结合起来用于VLN。我们发现,在仅有少量专家轨迹(10-100条)可用的场景下,我们的方法优于依赖视觉特征的基线,展示了语言作为导航感知表征的潜力。

英文摘要:

We explore the use of language as a perceptual representation for vision-and-language navigation (VLN), with a focus on low-data settings. Our approach uses off-the-shelf vision systems for image captioning and object detection to convert an agent's egocentric panoramic view at each time step into natural language descriptions. We then finetune a pretrained language model to select an action, based on the current view and the trajectory history, that would best fulfill the navigation instructions. In contrast to the standard setup which adapts a pretrained language model to work directly with continuous visual features from pretrained vision models, our approach instead uses (discrete) language as the perceptual representation. We explore several use cases of our language-based navigation (LangNav) approach on the R2R VLN benchmark: generating synthetic trajectories from a prompted language model (GPT-4) with which to finetune a smaller language model; domain transfer where we transfer a policy learned on one simulated environment (ALFRED) to another (more realistic) environment (R2R); and combining both vision- and language-based representations for VLN. Our approach is found to improve upon baselines that rely on visual features in settings where only a few expert trajectories (10-100) are available, demonstrating the potential of language as a perceptual representation for navigation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑