移动以理解3D场景：打通视觉定位与探索以实现高效通用的具身导航
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
AI总结：
该研究提出统一框架MTU3D，融合主动感知与3D视觉-语言学习，通过三项创新实现高效具身导航，在多基准上成功率优于现有方法，支持多模态输入，凸显打通视觉定位与探索对具身智能的价值。
摘要：
具身场景理解不仅需要理解已观测到的视觉空间信息，还需要确定在3D物理世界中下一步的探索位置。现有的3D视觉-语言（3D-VL）模型主要聚焦于在3D重建（如网格、点云）得到的静态观测中对物体进行视觉定位，但缺乏主动感知和探索环境的能力。为解决这一局限，我们提出Move to Understand（MTU3D，即“移动以理解”）——一个将主动感知与3D视觉-语言学习相融合的统一框架，使具身智能体能够高效探索并理解环境。该框架通过三项关键创新实现：1）基于在线查询的表示学习，可直接从RGB-D帧构建空间记忆，无需显式3D重建。2）视觉定位与探索的统一目标，将未探索位置表示为前沿查询，联合优化物体视觉定位与前沿选择。3）端到端轨迹学习，结合了视觉-语言-探索（Vision-Language-Exploration，VLE）预训练，预训练数据为从模拟与真实世界RGB-D序列中采集的百万条多样化轨迹。在多个具身导航与问答基准上的广泛评估显示，MTU3D在HM3D-OVON、GOAT-Bench、SG3D和A-EQA数据集上的成功率，分别优于当前最优的强化学习与模块化导航方法14%、23%、9%和2%。MTU3D的多功能性使其支持多种输入模态的导航，包括类别、语言描述和参考图像。这些发现凸显了打通视觉定位与探索环节对具身智能的重要性。
英文摘要：
Embodied scene understanding requires not only comprehending visual-spatial information that has been observed but also determining where to explore next in the 3D physical world. Existing 3D Vision-Language (3D-VL) models primarily focus on grounding objects in static observations from 3D reconstruction, such as meshes and point clouds, but lack the ability to actively perceive and explore their environment. To address this limitation, we introduce \underline{\textbf{M}}ove \underline{\textbf{t}}o \underline{\textbf{U}}nderstand (\textbf{\model}), a unified framework that integrates active perception with \underline{\textbf{3D}} vision-language learning, enabling embodied agents to effectively explore and understand their environment. This is achieved by three key innovations: 1) Online query-based representation learning, enabling direct spatial memory construction from RGB-D frames, eliminating the need for explicit 3D reconstruction. 2) A unified objective for grounding and exploring, which represents unexplored locations as frontier queries and jointly optimizes object grounding and frontier selection. 3) End-to-end trajectory learning that combines \textbf{V}ision-\textbf{L}anguage-\textbf{E}xploration pre-training over a million diverse trajectories collected from both simulated and real-world RGB-D sequences. Extensive evaluations across various embodied navigation and question-answering benchmarks show that MTU3D outperforms state-of-the-art reinforcement learning and modular navigation approaches by 14\%, 23\%, 9\%, and 2\% in success rate on HM3D-OVON, GOAT-Bench, SG3D, and A-EQA, respectively. \model's versatility enables navigation using diverse input modalities, including categories, language descriptions, and reference images. These findings highlight the importance of bridging visual grounding and exploration for embodied intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
