基于视觉语言模型的视觉语言导航模型依赖什么：解释与引导策略行为

中文摘要 AI 辅助

现代视觉语言导航（VLN）模型主要依赖预训练的大规模视觉语言模型（VLM）来预测导航动作。虽然语言指令与视觉观察的融合实现了多模态推理，但这也模糊了信息如何在模态间路由以及何种机制驱动导航决策。因此，目前尚不清楚VLN模型是否将其预测基于相关的语义线索，或能否跟踪任务进度。在本工作中，我们研究了VLN模型的可解释性和可引导性。我们使用基于干预的度量方法，衡量视觉观察、指令和视觉记忆如何因果性地影响导航决策。我们的结果表明，这些导航策略对所有输入模态都敏感，并不依赖于单一模态。我们进一步表明，这些智能体编码了导航进度，并从其VLM骨干中保留了语义结构，从而能够通过内部激活实现概念级别的引导。最后，我们提取抽象行为的激活向量，将其零样本迁移到分布外的真实世界场景中，在无需额外微调的情况下提升性能。

英文摘要

Modern Vision-Language Navigation (VLN) models rely mostly on pre-trained large Vision-Language Models (VLMs) to predict navigation actions. While this fusion of language instructions and visual observations allows multimodal reasoning, it obscures how information is routed across modalities or what mechanisms drive navigation decisions. Thus, it remains unclear whether VLN models ground their predictions in relevant semantic cues or can track task progress. In this work, we study the interpretability and steerability of VLN models. We use intervention-based metrics that measure how visual observations, instructions, and visual memory causally influence navigation decisions. Our results show that these navigation policies are sensitive to all input modalities and do not depend on a single one. We further show that these agents encode navigation progress and retain semantic structure from their VLM backbones, enabling concept-level steering through internal activations. Finally, we extract activation vectors for abstract behaviors to transfer them zero-shot to out-of-distribution real-world scenarios, improving performance without additional fine-tuning.

发表机构

• University of Technology Nuremberg（纽伦堡工业大学）
• ETH Zürich（苏黎世联邦理工学院）
• Inria, Ecole Normale Supérieure, CNRS, PSL Research University（法国国家信息与自动化研究所，巴黎高等师范学校，法国国家科学研究中心，巴黎文理研究大学）

机构由 AI 辅助整理，请以论文原文为准。