arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.12908cs.RO
收藏

机器人操作是视觉到几何映射(f(v)→G):超越语言和视频模型的视觉-几何骨干

Robotic Manipulation is Vision-to-Geometry Mapping: Vision-Geometry Backbones over Language and Video Models

Sun Yat-sen University(中山大学);Guangdong Key Laboratory of Big Data Analysis(大数据分析与处理广东省重点实验室);X-Era AI Lab(X-Era人工智能实验室);AMAP, Alibaba(阿里妈妈实验室);Guangdong University of Technology(广东工业大学)

Zijian Song, Qichang Li, Jiawei Zhou, Zhenlong Yuan, Tianshui Chen, Liang Lin, Guangrun Wang

AI总结:

本文提出基于视觉-几何骨干的机器人操作方法,通过直接利用预训练的3D表示生成动作,优于传统语言或视频模型,在精确操作和零样本泛化中表现更优。

摘要:

机器人操作本质上是视觉到几何映射(f(v)→G)的问题。物理动作本质上由三维位置和空间关系等几何属性定义。因此,我们认为通用机器人控制的基础应是视觉-几何骨干,而非广泛采用的视觉-语言或视频模型。传统VLA和视频预测模型依赖于在大规模2D图像-文本或时间像素数据上预训练的骨干。虽然有效,但它们的表示主要受语义概念或2D先验的影响,这与物理操作所需的精确3D几何性质不一致。受此启发,我们提出了视觉-几何-动作(VGA)模型,该模型直接将预训练的原生3D表示作为条件,生成动作。具体而言,VGA用预训练的3D世界模型取代传统语言或视频骨干,建立无缝的视觉到几何映射,将视觉输入直接转化为物理动作。为进一步增强几何一致性,我们引入了渐进体积调制模块并采用联合训练策略。大量实验验证了本方法的有效性。在模拟基准测试中,VGA优于顶级VLA基线,包括π_0.5和GeoVLA,证明了其在精确操作中的优越性。更重要的是,VGA在现实部署中表现出显著的零样本泛化能力,在未见过的视角上持续优于π_0.5。这些结果表明,基于原生3D表示而非通过语言或2D视频先验进行转换,是实现通用物理智能的有前途方向。

英文摘要:

At its core, robotic manipulation is a problem of vision-to-geometry mapping ($f(v) \rightarrow G$). Physical actions are fundamentally defined by geometric properties like 3D positions and spatial relationships. Consequently, we argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models. Conventional VLA and video-predictive models rely on backbones pretrained on large-scale 2D image-text or temporal pixel data. While effective, their representations are largely shaped by semantic concepts or 2D priors, which do not intrinsically align with the precise 3D geometric nature required for physical manipulation. Driven by this insight, we propose the Vision-Geometry-Action (VGA) model, which directly conditions action generation on pretrained 3D representations. Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping that translates visual inputs directly into physical actions. To further enhance geometric consistency, we introduce Progressive Volumetric Modulation and jointly train action and 3D property prediction to preserve geometric representations. Extensive experiments validate the effectiveness of our approach. Across simulation benchmarks, VGA outperforms leading VLA, 3D-VLA, and WAM baselines, including $π_{0.5}$, OpenVLA-OFT, GeoVLA, and Motus. In real-world deployments, VGA surpasses $π_{0.5}$ under unseen viewpoints and accurately follows language instructions for target grasping. These results highlight that operating on native 3D representations, rather than relying primarily on language or video priors, offers a promising direction toward generalizable physical intelligence. Project page: https://hcplab-sysu.github.io/VisionGeometryActionModel.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑