arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.17439cs.ROcs.AIcs.CVcs.LG

从空间到动作:在空间先验基础上构建视觉-语言-动作模型

From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors

ByteDance Seed(字节跳动种子);NUS(新加坡国立大学);NTU(国立技术大学);THU(清华大学);SMU(南方大学)

Zhengshen Zhang, Hao Li, Yalun Dai, Zhengbang Zhu, Lei Zhou, Chenchen Liu, Dong Wang, Francis E. H. Tay, Sijin Chen, Ziwei Liu, Yuxiao Liu, Xinghang Li, Pan Zhou

AI总结:

FALCON通过引入丰富的3D空间标记,改进了视觉-语言-动作模型的空间表示、模态可转移性和对齐能力,在多个基准和现实任务中取得最佳性能。

摘要:

现有的视觉-语言-动作(VLA)模型在三维真实世界中发挥作用,但通常基于二维编码器构建,留下空间推理的空白,限制了泛化能力和适应性。最近的3D整合技术用于VLA要么需要专用传感器并在跨模态转移时表现不佳,要么注入弱提示,缺乏几何信息并降级了视觉-语言对齐。在本文中,我们引入FALCON(从空间到动作),一种新的范式,将丰富的3D空间标记注入动作头中。FALCON利用空间基础模型,仅通过RGB即可提供强大的几何先验,并包含一个具身体验空间模型,可以可选地融合深度或姿态以获得更高的保真度,无需重新训练或架构更改。为了保持语言推理,空间标记由增强的空间动作头消耗,而不是被连接到视觉-语言骨干中。这些设计使FALCON能够解决空间表示、模态可转移性和对齐的限制。在三个模拟基准和十一个现实任务的综合评估中,我们提出的FALCON实现了最先进的性能,始终优于竞争基线,并在杂乱、空间提示条件和物体尺寸和高度变化下保持稳健。

英文摘要:

Existing vision-language-action (VLA) models act in 3D real-world but are typically built on 2D encoders, leaving a spatial reasoning gap that limits generalization and adaptability. Recent 3D integration techniques for VLAs either require specialized sensors and transfer poorly across modalities, or inject weak cues that lack geometry and degrade vision-language alignment. In this work, we introduce FALCON (From Spatial to Action), a novel paradigm that injects rich 3D spatial tokens into the action head. FALCON leverages spatial foundation models to deliver strong geometric priors from RGB alone, and includes an Embodied Spatial Model that can optionally fuse depth, or pose for higher fidelity when available, without retraining or architectural changes. To preserve language reasoning, spatial tokens are consumed by a Spatial-Enhanced Action Head rather than being concatenated into the vision-language backbone. These designs enable FALCON to address limitations in spatial representation, modality transferability, and alignment. In comprehensive evaluations across three simulation benchmarks and eleven real-world tasks, our proposed FALCON achieves state-of-the-art performance, consistently surpasses competitive baselines, and remains robust under clutter, spatial-prompt conditioning, and variations in object scale and height.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑