D3D-VLP：动态3D视觉-语言-规划模型用于具身接地与导航
D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation
AI总结：
D3D-VLP通过动态3D链式思维和协同学习策略，实现具身接地与导航的高效统一，提升多任务导航性能。
摘要：
具身代理面临一个关键困境：端到端模型缺乏可解释性且缺乏显式的3D推理，而模块化系统则忽略了跨组件的依赖关系和协同效应。为弥合这一差距，我们提出了动态3D视觉-语言-规划模型（D3D-VLP）。我们的模型引入了两个关键创新：1）动态3D链式思维（3D CoT），它在单一的3D-VLM和CoT流程中统一了规划、接地、导航和问答；2）协同学习从碎片化监督（SLFS）策略，利用掩码自回归损失从大规模且部分标注的混合数据中学习。这使得不同的CoT组件可以相互强化并隐式地监督彼此。为此，我们构建了一个大规模数据集，包含来自5000个真实扫描和20000个合成场景的1000万混合样本，这些样本与在线学习方法如强化学习和DAgger兼容。我们的D3D-VLP在多个基准上实现了最先进的结果，包括视觉-语言导航（R2R-CE、REVERIE-CE、NavRAG-CE）、目标导航（HM3D-OVON）以及任务导向的序列接地与导航（SG3D）。现实中的移动机械手实验进一步验证了其有效性。
英文摘要：
Embodied agents face a critical dilemma that end-to-end models lack interpretability and explicit 3D reasoning, while modular systems ignore cross-component interdependencies and synergies. To bridge this gap, we propose the Dynamic 3D Vision-Language-Planning Model (D3D-VLP). Our model introduces two key innovations: 1) A Dynamic 3D Chain-of-Thought (3D CoT) that unifies planning, grounding, navigation, and question answering within a single 3D-VLM and CoT pipeline; 2) A Synergistic Learning from Fragmented Supervision (SLFS) strategy, which uses a masked autoregressive loss to learn from massive and partially-annotated hybrid data. This allows different CoT components to mutually reinforce and implicitly supervise each other. To this end, we construct a large-scale dataset with 10M hybrid samples from 5K real scans and 20K synthetic scenes that are compatible with online learning methods such as RL and DAgger. Our D3D-VLP achieves state-of-the-art results on multiple benchmarks, including Vision-and-Language Navigation (R2R-CE, REVERIE-CE, NavRAG-CE), Object-goal Navigation (HM3D-OVON), and Task-oriented Sequential Grounding and Navigation (SG3D). Real-world mobile manipulation experiments further validate the effectiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
