TraceVLA:视觉轨迹提示增强通用机器人策略的时空感知能力
TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies
AI总结:
本文提出视觉轨迹提示方法,通过在15万条操作轨迹数据集上微调OpenVLA得到TraceVLA模型,显著提升机器人策略的时空感知能力,在仿真和真实机器人任务中均优于基线并具备良好泛化性。
摘要:
尽管在大规模机器人数据集上预训练的大型视觉-语言-动作(VLA)模型为机器人学习提供了有前景的通用策略,但它们在交互式机器人中的时空动态方面仍然存在困难,导致其在处理操作等复杂任务时效果不佳。在本工作中,我们提出了视觉轨迹提示(visual trace prompting),这是一种简单而有效的方法,通过以视觉方式编码状态-动作轨迹来提升VLA模型的时空感知能力,从而改善动作预测。我们通过在自采集的包含15万条机器人操作轨迹的数据集上使用视觉轨迹提示对OpenVLA进行微调,开发了新的TraceVLA模型。TraceVLA在SimplerEnv的137种配置和实体WidowX机器人上的4项任务中的评估结果表明其具有最先进的性能,在SimplerEnv上比OpenVLA高出10%,在真实机器人任务上性能提升3.5倍,并在不同本体和场景中表现出鲁棒的泛化能力。为了进一步验证我们方法的有效性和通用性,我们提出了一个基于4B Phi-3-Vision的紧凑VLA模型,该模型在Open-X-Embodiment上预训练并在我们的数据集上微调,其性能可与7B OpenVLA基线相媲美,同时显著提高了推理效率。
英文摘要:
Although large vision-language-action (VLA) models pretrained on extensive robot datasets offer promising generalist policies for robotic learning, they still struggle with spatial-temporal dynamics in interactive robotics, making them less effective in handling complex tasks, such as manipulation. In this work, we introduce visual trace prompting, a simple yet effective approach to facilitate VLA models' spatial-temporal awareness for action prediction by encoding state-action trajectories visually. We develop a new TraceVLA model by finetuning OpenVLA on our own collected dataset of 150K robot manipulation trajectories using visual trace prompting. Evaluations of TraceVLA across 137 configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and 3.5x on real-robot tasks and exhibiting robust generalization across diverse embodiments and scenarios. To further validate the effectiveness and generality of our method, we present a compact VLA model based on 4B Phi-3-Vision, pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B OpenVLA baseline while significantly improving inference efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。