arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.23721cs.ROcs.CV
收藏

StemVLA:一种融合未来3D空间几何知识与4D历史表示的开源视觉-语言-动作模型

StemVLA:An Open-Source Vision-Language-Action Model with Future 3D Spatial Geometry Knowledge and 4D Historical Representation

Ricoh Software Research Center(理光软件研究中心);YZH Engineering Technology (Beijing) Co., Ltd.(易智赫工程技术(北京)有限公司)

Jiasong Xiao, Yutao She, Kai Li, Yuyang Sha, Ziang Cheng

AI总结:

提出StemVLA框架,通过显式建模未来3D空间结构和历史4D时空表示,提升机器人操作任务中的空间推理与长时决策能力,在LIBERO基准上平均准确率达92.0%。

摘要:

视觉-语言-动作(VLA)模型整合视觉观测和语言指令来预测机器人动作,在操作任务中展现出有前景的泛化能力。然而,现有方法大多依赖于从2D视觉输入到动作序列的直接映射,未显式建模底层3D空间结构或时间世界动态。这种表示可能限制动态环境中的空间推理和长时决策。为解决这一局限,我们提出StemVLA,一种新颖框架,显式地将面向未来的3D空间知识和历史4D时空表示融入动作预测。首先,StemVLA不仅依赖观测图像,还预测结构化的未来3D空间几何世界知识,使模型能够预判即将到来的场景几何和物体配置。其次,为捕捉时间一致性和运动动态,我们将历史图像帧输入预训练的视频几何变换器骨干网络,提取隐式3D世界表示,并通过时间注意力模块(称为VideoFormer [20])跨时间聚合,形成统一的4D历史时空表示。通过联合建模2D观测、预测的3D未来结构和聚合的4D时间动态,StemVLA实现了对机器人操作更全面的世界理解。在仿真中的大量实验表明,StemVLA在LIBERO子集上平均准确率达92.0%,在长时LIBERO-Long子集上达86.0%。

英文摘要:

Vision-language-action (VLA) models integrate visual observations and language instructions to predict robot actions, demonstrating promising generalization in manipulation tasks. However, most existing approaches primarily rely on direct mappings from 2D visual inputs to action sequences, without explicitly modeling the underlying 3D spatial structure or temporal world dynamics. Such representations may limit spatial reasoning and long-horizon decision-making in dynamic environments. To address this limitation, we propose StemVLA, a novel framework that explicitly incorporates both future-oriented 3D spatial knowledge and historical 4D spatiotemporal representations into action prediction. First, instead of relying solely on observed images, StemVLA forecasts structured 3D future spatial-geometric world knowledge, enabling the model to anticipate upcoming scene geometry and object configurations. Second, to capture temporal consistency and motion dynamics, we feed historical image frames into a pretrained video-geometry transformer backbone to extract implicit 3D world representations, and further aggregate them across time using a temporal attention module, termed VideoFormer [20], forming a unified 4D historical spatiotemporal representation. By jointly modeling 2D observations, predicted 3D future structure, and aggregated 4D temporal dynamics, StemVLA enables more comprehensive world understanding for robot manipulation. Extensive experiments in simulation demonstrate that Stem-VLA achieves an average accuracy of 92.0% across the LIBERO subsets, and 86.0% on the long-horizon LIBERO-Long subset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑