CMMR-VLN:通过持续多模态记忆检索实现视觉与语言导航
CMMR-VLN: Vision-and-Language Navigation via Continual Multimodal Memory Retrieval
AI总结:
CMMR-VLN通过引入持续多模态记忆检索机制,提升视觉与语言导航任务中对先前经验的选择性利用能力,显著提高导航成功率。
摘要:
尽管大语言模型(LLMs)被引入视觉与语言导航(VLN)以提高指令理解和泛化能力,但现有的基于LLM的VLN缺乏选择性回忆和利用相关先前经验的能力,限制了其在长视野和不熟悉场景中的表现。在本工作中,我们提出了CMMR-VLN(基于持续多模态记忆检索的VLN),一种赋予LLM代理结构化记忆和反思能力的VLN框架。具体而言,CMMR-VLN构建了一个由全景视觉图像和显著地标索引的多模态经验记忆,以在导航过程中检索相关经验,引入了一个检索增强的生成管道以模仿有经验的导航者如何利用先前知识,以及采用基于反思的记忆更新策略,该策略选择性地存储完整的成功路径和失败案例中的关键初始错误。全面测试显示,与NavGPT、MapGPT和DiscussNav在模拟和实际测试中的平均成功率分别提高了52.9%、20.9%和20.9%,以及200%、50%和50%,阐明了CMMR-VLN作为基础VLN框架的巨大潜力。
英文摘要:
Although large language models (LLMs) are introduced into vision-and-language navigation (VLN) to improve instruction comprehension and generalization, existing LLM- based VLN lacks the ability to selectively recall and use relevant priori experiences to help navigation tasks, limiting their performance in long-horizon and unfamiliar scenarios. In this work, we propose CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities. Specifically, the CMMR-VLN constructs a multimodal experi- ence memory indexed by panoramic visual images and salient landmarks to retrieve relevant experiences during navigation, introduces a retrieved-augmented generation pipeline to mimick how experienced human navigators leverage priori knowledge, and incorporates a reflection-based memory update strategy that selectively stores complete successful paths and the key initial mistake in failure cases. Comprehensive tests illustrate average success rate improvements of 52.9%, 20.9% and 20.9%, and 200%, 50% and 50% over the NavGPT, the MapGPT, and the DiscussNav in simulation and real tests, respectively eluci- dating the great potential of the CMMR-VLN as a backbone VLN framework.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。