arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.24393cs.RO

3D-Mix用于VLA:一种用于将基于VGGT的3D信息整合到视觉-语言-动作模型中的即插即用模块

3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models

HIT(哈尔滨工业大学);ZGCA(中钢集团自动化研究院);ZGCI(中钢集团信息研究院);HUST(华中科技大学);HKUST(GZ)(香港科技大学(广州));BUAA(北京航空航天大学);ECNU(华东师范大学);DeepCybo

Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Haishan Liu, Changti Wu, Hang Yuan, Bailing Wang, Cong Huang, Kai Chen

AI总结:

本文提出3D-Mix模块,通过语义条件门控融合机制提升视觉-语言-动作模型的3D感知能力,在标准化基准测试中实现最佳性能。

摘要:

视觉-语言-动作(VLA)模型利用多模态大语言模型(MLLMs)进行机器人控制,但最近研究表明,由于训练主要基于2D数据,MLLMs在空间智能方面存在局限,导致在操纵任务中的3D感知不足。尽管近期方法整合了专门的3D视觉模型如VGGT以增强空间理解,但它们采用多样的整合机制而缺乏系统研究,使最佳融合策略仍不明确。我们进行了全面的试点研究,比较了九种VGGT整合方案,并发现基于任务上下文的语义条件门控融合,通过自适应平衡2D语义和3D几何特征,在我们的试点研究中在所有九种评估融合方案中表现最强。我们提出了3D-Mix,这是一种即插即用模块,可以整合到不同的VLA架构(GR00T风格和π风格)中,而无需修改现有的MLLM或动作专家组件。在六个MLLM系列(九种模型变体,2B-8B参数)上进行的实验显示,3D-Mix在SIMPLER和LIBERO上实现了一致的性能提升,在所有九种GR00T风格变体上,平均在out-of-domain(OOD)SIMPLER基准上提升了+7.0%,建立了增强VLA系统空间智能的系统方法。

英文摘要:

Vision-Language-Action (VLA) models leverage Multimodal Large Language Models (MLLMs) for robotic control, but recent studies reveal that MLLMs exhibit limited spatial intelligence due to training predominantly on 2D data, resulting in inadequate 3D perception for manipulation tasks. While recent approaches incorporate specialized 3D vision models such as VGGT to enhance spatial understanding, they employ diverse integration mechanisms without systematic investigation, leaving the optimal fusion strategy unclear. We conduct a comprehensive pilot study comparing nine VGGT integration schemes on standardized benchmarks and find that semantic-conditioned gated fusion, which adaptively balances 2D semantic and 3D geometric features based on task context, achieved the strongest performance among all nine evaluated fusion schemes in our pilot study. We present 3D-Mix, a plug-and-play module that integrates into diverse VLA architectures (GR00T-style and $π$-style) without modifying existing MLLM or action expert components. Experiments across six MLLM series (nine model variants, 2B--8B parameters) on SIMPLER and LIBERO show that 3D-Mix delivers consistent performance gains, averaging +7.0% on the out-of-domain (OOD) SIMPLER benchmark across all nine GR00T-style variants, establishing a principled approach for enhancing spatial intelligence in VLA systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑