arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.09280cs.CV
收藏

用于视觉语言导航的增强结构化状态进化

Reinforced Structured State-Evolution for Vision-Language Navigation

Beihang University(北京航空航天大学);Hangzhou Innovation Institute, Beihang University(北京航空航天大学杭州创新研究院)

Jinyu Chen, Chen Gao, Erli Meng, Qiong Zhang, Si Liu

摘要:

视觉和语言导航(VLN)任务需要一个具体的代理导航到一个远程位置以下的自然语言指令。先前的方法通常采用序列模型(例如,Transformer和LSTM)作为导航器。在这样的范例中,序列模型通过维护的导航状态来预测每一步的动作,该导航状态通常表示为一维向量。然而,关键的导航线索(即,对象级环境布局)被丢弃,因为所维护的向量基本上是非结构化的。在本文中,我们提出了一种新的结构化状态演化(SEvol)模型,以有效地保持环境布局的线索VLN。具体来说,我们利用基于图形的功能来表示导航状态,而不是基于矢量的状态。因此,我们设计了一个强化布局线索矿工(RLM)挖掘和检测最关键的布局图,通过定制的强化学习策略的长期导航。此外,结构化的进化模块(SEM)提出了在导航过程中保持结构化的基于图的状态,其中状态逐渐进化,学习对象级的时空关系。在R2R和R4R数据集上的实验表明,所提出的SEvol模型大大提高了VLN模型的性能,例如,在R2R测试集上,NvEM的绝对SPL准确度为+3%,EnvDrop的绝对SPL准确度为+8%。

英文摘要:

Vision-and-language Navigation (VLN) task requires an embodied agent to navigate to a remote location following a natural language instruction. Previous methods usually adopt a sequence model (e.g., Transformer and LSTM) as the navigator. In such a paradigm, the sequence model predicts action at each step through a maintained navigation state, which is generally represented as a one-dimensional vector. However, the crucial navigation clues (i.e., object-level environment layout) for embodied navigation task is discarded since the maintained vector is essentially unstructured. In this paper, we propose a novel Structured state-Evolution (SEvol) model to effectively maintain the environment layout clues for VLN. Specifically, we utilise the graph-based feature to represent the navigation state instead of the vector-based state. Accordingly, we devise a Reinforced Layout clues Miner (RLM) to mine and detect the most crucial layout graph for long-term navigation via a customised reinforcement learning strategy. Moreover, the Structured Evolving Module (SEM) is proposed to maintain the structured graph-based state during navigation, where the state is gradually evolved to learn the object-level spatial-temporal relationship. The experiments on the R2R and R4R datasets show that the proposed SEvol model improves VLN models' performance by large margins, e.g., +3% absolute SPL accuracy for NvEM and +8% for EnvDrop on the R2R test set.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑