GeoVLA：在视觉-语言-动作模型中引入3D表示
GeoVLA: Empowering 3D Representations in Vision-Language-Action Models
AI总结：
针对现有VLA模型依赖2D视觉输入导致空间感知受限的问题，提出GeoVLA框架，通过引入3D点云几何嵌入与空间感知动作专家，显著提升了机器人在仿真与真实环境中的操作精度和鲁棒性。
摘要：
视觉-语言-动作（VLA）模型已成为使机器人能够遵循语言指令并预测相应动作的一种极具前景的方法。然而，当前的VLA模型主要依赖2D视觉输入，忽略了3D物理世界中丰富的几何信息，这限制了其空间感知能力与适应性。在本文中，我们提出了GeoVLA，这是一个有效整合3D信息以推进机器人操作的全新VLA框架。它使用视觉-语言模型（VLM）来处理图像和语言指令，提取融合的视觉-语言嵌入。同时，它将深度图转换为点云，并采用一个定制的点编码器（称为Point Embedding Network）独立生成3D几何嵌入。随后，这些生成的嵌入被拼接起来，并由我们提出的具备空间感知能力的动作专家（称为3D-enhanced Action Expert）进行处理，该专家结合来自不同传感器模态的信息以生成精确的动作序列。通过在仿真和真实环境中的大量实验，GeoVLA展现出了优越的性能与鲁棒性。它在LIBERO和ManiSkill2仿真基准中取得了最先进的结果，并在需要高度适应性、尺度感知和视角不变性的真实世界任务中表现出卓越的鲁棒性。
英文摘要：
Vision-Language-Action (VLA) models have emerged as a promising approach for enabling robots to follow language instructions and predict corresponding actions. However, current VLA models mainly rely on 2D visual inputs, neglecting the rich geometric information in the 3D physical world, which limits their spatial awareness and adaptability. In this paper, we present GeoVLA, a novel VLA framework that effectively integrates 3D information to advance robotic manipulation. It uses a vision-language model (VLM) to process images and language instructions,extracting fused vision-language embeddings. In parallel, it converts depth maps into point clouds and employs a customized point encoder, called Point Embedding Network, to generate 3D geometric embeddings independently. These produced embeddings are then concatenated and processed by our proposed spatial-aware action expert, called 3D-enhanced Action Expert, which combines information from different sensor modalities to produce precise action sequences. Through extensive experiments in both simulation and real-world environments, GeoVLA demonstrates superior performance and robustness. It achieves state-of-the-art results in the LIBERO and ManiSkill2 simulation benchmarks and shows remarkable robustness in real-world tasks requiring height adaptability, scale awareness and viewpoint invariance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
