arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1906.00347cs.CL
收藏

你在看吗?从Grounding到视觉和语言导航中的多模态

Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation

University of California, Berkeley(加州大学伯克利分校);Boston University(波士顿大学)

Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, Kate Saenko

摘要:

视觉和语言导航(VLN)需要基础指令,如“右转,停在门口”,在视觉环境中的路线。实际的基础可以通过多种形式将语言与环境联系起来,例如,“在门口停下来”可能会成为视觉对象,而“右转”可能只依赖于路线的几何结构。我们调查的自然语言经验的理由下两个最近的国家的最先进的VLN模型。令人惊讶的是,我们发现视觉特征实际上可能会伤害这些模型:仅使用路线结构的模型,消融视觉特征,在基准Room-to-Room数据集上看不见的新环境中优于其视觉对应物。为了更好地使用所有可用的模态,我们建议将接地过程分解为一组可以访问不同模态(包括目标检测)的专家模型,并在预测时将其集成,从而提高VLN任务中最先进模型的性能。

英文摘要:

Vision-and-Language Navigation (VLN) requires grounding instructions, such as "turn right and stop at the door", to routes in a visual environment. The actual grounding can connect language to the environment through multiple modalities, e.g. "stop at the door" might ground into visual objects, while "turn right" might rely only on the geometric structure of a route. We investigate where the natural language empirically grounds under two recent state-of-the-art VLN models. Surprisingly, we discover that visual features may actually hurt these models: models which only use route structure, ablating visual features, outperform their visual counterparts in unseen new environments on the benchmark Room-to-Room dataset. To better use all the available modalities, we propose to decompose the grounding procedure into a set of expert models with access to different modalities (including object detections) and ensemble them at prediction time, improving the performance of state-of-the-art models on the VLN task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑