面向视觉语言导航的接地实体-地标自适应预训练
Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation
AI总结：
针对视觉语言导航中实体级细粒度跨模态对齐不足的问题，提出GELA自适应预训练范式，构建GEL-R2R数据集并设计三个预训练目标，在R2R和CVDN基准上取得最优结果。
摘要：
跨模态对齐是视觉语言导航（VLN）的核心挑战之一。现有研究大多聚焦于将全局指令或单条子指令映射到对应轨迹，却很少关注实现实体级细粒度对齐这一关键问题。为解决该问题，我们提出一种面向VLN任务的新型接地实体-地标自适应（GELA）预训练范式。为实现这一自适应预训练范式，我们首先在Room-to-Room（R2R）数据集中引入人工标注的接地实体-地标数据，命名为GEL-R2R。此外，我们采用三个接地实体-地标自适应预训练目标：1）实体短语预测，2）地标边界框预测，3）实体-地标语义对齐，显式监督实体短语与环境地标之间细粒度跨模态对齐的学习。最后，我们在两个下游基准上验证模型：带描述性指令的VLN（R2R）和带对话指令的VLN（CVDN）。综合实验表明，我们的GELA模型在两项任务上均取得了最优结果，证明了其有效性和泛化性。
英文摘要：
Cross-modal alignment is one key challenge for Vision-and-Language Navigation (VLN). Most existing studies concentrate on mapping the global instruction or single sub-instruction to the corresponding trajectory. However, another critical problem of achieving fine-grained alignment at the entity level is seldom considered. To address this problem, we propose a novel Grounded Entity-Landmark Adaptive (GELA) pre-training paradigm for VLN tasks. To achieve the adaptive pre-training paradigm, we first introduce grounded entity-landmark human annotations into the Room-to-Room (R2R) dataset, named GEL-R2R. Additionally, we adopt three grounded entity-landmark adaptive pre-training objectives: 1) entity phrase prediction, 2) landmark bounding box prediction, and 3) entity-landmark semantic alignment, which explicitly supervise the learning of fine-grained cross-modal alignment between entity phrases and environment landmarks. Finally, we validate our model on two downstream benchmarks: VLN with descriptive instructions (R2R) and dialogue instructions (CVDN). The comprehensive experiments show that our GELA model achieves state-of-the-art results on both tasks, demonstrating its effectiveness and generalizability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
