arXiv 2603.092592026-03-10cs.CVcs.RO

从网络视频中隐式几何表示的视觉-语言导航

Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos

Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence（计算机视觉系，Mohamed Bin Zayed人工智能大学）；School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学）；Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）

Mingfei Han, Haihong Hao, Liang Ma, Kamila Zhumakhanova, Ekaterina Radionova, Jingyi Zhang, Xiaojun Chang, Xiaodan Liang, Ivan Laptev

备注：: Extension of CVPR 2025 RoomTour3D with implicit geometric representations

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出基于网络视频的隐式几何表示方法，用于视觉-语言导航，通过提升数据利用率和鲁棒性，实现更广泛的应用。

摘要：

视觉-语言导航（VLN）长期以来受到模拟器定制数据集多样性有限和可扩展性差的限制，这些数据集无法捕捉真实环境的复杂性。为克服这一限制，我们引入了一个基于网络房间游览视频的大型视频-指令框架，使智能体能够从自然人类行走示范中学习，从而在多样且真实的室内环境中进行学习。与现有数据集不同，我们的框架整合了开放式描述增强轨迹和动作增强轨迹，这些轨迹在3D中重建，提供了更丰富的空间和语义监督。本工作的一项关键扩展是引入隐式几何表示，该方法直接从RGB帧中提取空间线索，而无需依赖脆弱的3D重建。这种方法显著提高了数据利用率，缓解了重建失败的问题，并解锁了之前无法使用的大量视频数据。在多个VLN基准（CVDN、SOON、R2R和REVERIE）上的全面实验表明，我们的方法不仅设定了新的最先进性能，还使开发鲁棒的零样本导航智能体成为可能。通过将大规模网络视频与隐式空间推理相结合，这项工作推动了具身导航朝着更可扩展、可推广和适用于现实世界解决方案的方向发展。

英文摘要：

Vision-and-Language Navigation (VLN) has long been constrained by the limited diversity and scalability of simulator-curated datasets, which fail to capture the complexity of real-world environments. To overcome this limitation, we introduce a large-scale video-instruction framework derived from web-based room tour videos, enabling agents to learn from natural human walking demonstrations in diverse, realistic indoor settings. Unlike existing datasets, our framework integrates both open-ended description-enriched trajectories and action-enriched trajectories reconstructed in 3D, providing richer spatial and semantic supervision. A key extension in this work is the incorporation of implicit geometry representations, which extract spatial cues directly from RGB frames without requiring fragile 3D reconstruction. This approach substantially improves data utilization, alleviates reconstruction failures, and unlocks large portions of previously unusable video data. Comprehensive experiments across multiple VLN benchmarks (CVDN, SOON, R2R, and REVERIE) demonstrate that our method not only sets new state-of-the-art performance but also enables the development of robust zero-shot navigation agents. By bridging large-scale web videos with implicit spatial reasoning, this work advances embodied navigation towards more scalable, generalizable, and real-world applicable solutions.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。