arXiv 2504.165162025-04-24cs.CVcs.AI

分层思考，动态行动：面向视觉与语言导航的分层多模态融合与推理

Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigation

City University of Hong Kong, Dongguan Campus（香港城市大学东莞校区）；The University of Melbourne（墨尔本大学）；Tsinghua University（清华大学）；Baidu Inc.（百度公司）；University of Chinese Academy of Science（中国科学院大学）；National University of Singapore（新加坡国立大学）

Junrong Yue, Yifan Zhang, Chuan Qin, Bo Li, Xiaomin Lie, Xinlei Yu, Wenxin Zhang, Zhendong Zhao

备注：: 11 pages, 4 figures, Submitted to ACM MM 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出多层级融合与推理架构（MFRA），通过分层融合视觉、语言与历史特征并动态推理，在REVERIE、R2R和SOON基准上超越现有方法，提升视觉语言导航决策准确性。

摘要：

视觉与语言导航（VLN）旨在使具身智能体能够遵循自然语言指令，在真实环境中到达目标位置。以往的方法往往依赖于全局场景表示或物体级特征，但这些方法不足以捕捉跨模态的复杂交互，而这对准确导航至关重要。在本文中，我们提出了一种多层级融合与推理架构（MFRA），以增强智能体对视觉观察、语言指令和导航历史进行推理的能力。具体而言，MFRA引入了一种分层融合机制，该机制跨多种模态聚合从低级视觉线索到高级语义概念的多层级特征。我们进一步设计了一个推理模块，该模块利用融合后的表示，通过指令引导的注意力和动态上下文整合来推断导航动作。通过选择性地捕获并组合相关的视觉、语言和时间信号，MFRA在复杂导航场景中提高了决策准确性。在包括REVERIE、R2R和SOON在内的基准VLN数据集上的大量实验表明，MFRA相较于最先进的方法取得了更优越的性能，验证了多层级模态融合对具身导航的有效性。

英文摘要：

Vision-and-Language Navigation (VLN) aims to enable embodied agents to follow natural language instructions and reach target locations in real-world environments. While prior methods often rely on either global scene representations or object-level features, these approaches are insufficient for capturing the complex interactions across modalities required for accurate navigation. In this paper, we propose a Multi-level Fusion and Reasoning Architecture (MFRA) to enhance the agent's ability to reason over visual observations, language instructions and navigation history. Specifically, MFRA introduces a hierarchical fusion mechanism that aggregates multi-level features-ranging from low-level visual cues to high-level semantic concepts-across multiple modalities. We further design a reasoning module that leverages fused representations to infer navigation actions through instruction-guided attention and dynamic context integration. By selectively capturing and combining relevant visual, linguistic, and temporal signals, MFRA improves decision-making accuracy in complex navigation scenarios. Extensive experiments on benchmark VLN datasets including REVERIE, R2R, and SOON demonstrate that MFRA achieves superior performance compared to state-of-the-art methods, validating the effectiveness of multi-level modal fusion for embodied navigation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。