arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05552cs.CV
收藏

眼见为实?利用视觉扰动增强视觉-语言导航

Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

The School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院)

Xuesong Zhang, Jia Li, Yunbo Xu, Zhenzhen Hu, Richang Hong

AI总结:

本文通过引入视觉扰动探究VLN模型对视觉内容的理解,发现简单分支扩展可提升导航效果,并提出通用多分支架构MBA,在R2R、REVERIE、SOON三个基准上达到或超越最先进结果。

摘要:

在具身环境中依据自然语言指令进行自主导航,对视觉-语言导航(VLN)智能体而言仍是一项挑战。尽管近期在学习多样且细粒度的视觉环境表征方面取得了进展,但其脆弱的性能提升可能并不能确凿地归因于视觉定位能力的增强,这一局限在相关视觉-语言任务中同样存在。本文初步探究了先进VLN模型是否真正理解其环境的视觉内容,方法是在环境中引入不同程度的视觉扰动,包括真实深度图像、扰动视图和随机噪声。令人意外的是,实验发现即使使用带噪声的视觉输入,简单的分支扩展反而能提升导航效果。受此启发,我们进一步提出了一种通用的多分支架构(MBA),用于深入研究分支数量和视觉质量的影响。所提出的MBA将基础智能体扩展为多分支变体,每个分支处理不同的视觉输入。该方法极其简单,且与基于拓扑结构的VLN智能体无关。在三个VLN基准(R2R、REVERIE、SOON)上的大量实验表明,在最优视觉排列下,我们的方法能够达到甚至超越最先进的结果。源代码已公开。

英文摘要:

Autonomous navigation guided by natural language instructions in embodied environments remains a challenge for vision-language navigation (VLN) agents. Although recent advancements in learning diverse and fine-grained visual environmental representations have shown promise, the fragile performance improvements may not conclusively attribute to enhanced visual grounding,a limitation also observed in related vision-language tasks. In this work, we preliminarily investigate whether advanced VLN models genuinely comprehend the visual content of their environments by introducing varying levels of visual perturbations. These perturbations include ground-truth depth images, perturbed views and random noise. Surprisingly, we experimentally find that simple branch expansion, even with noisy visual inputs, paradoxically improves the navigational efficacy. Inspired by these insights, we further present a versatile Multi-Branch Architecture (MBA) designed to delve into the impact of both the branch quantity and visual quality. The proposed MBA extends a base agent into a multi-branch variant, where each branch processes a different visual input. This approach is embarrassingly simple yet agnostic to topology-based VLN agents. Extensive experiments on three VLN benchmarks (R2R, REVERIE, SOON) demonstrate that our method with optimal visual permutations matches or even surpasses state-of-the-art results. The source code is available at here.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑