arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.12322cs.ROcs.AI
收藏

ForeAct: 通过高效的视觉前瞻性规划控制您的VLA

ForeAct: Steering Your VLA with Efficient Visual Foresight Planning

MIT(麻省理工学院);NVIDIA(英伟达);Caltech(加州理工学院)

Zhuoyang Zhang, Shang Yang, Qinghao Hu, Luke J. Huang, James Hou, Yufei Sun, Yao Lu, Song Han

AI总结:

ForeAct通过高效的视觉前瞻性规划提升VLA在开放环境中的执行精度与泛化能力。

摘要:

视觉-语言-动作(VLA)模型将高级语言指令转换为具体的可执行动作,这在开放世界环境中尤其具有挑战性。我们提出了视觉前瞻性规划(ForeAct),一种通用且高效的规划器,它通过想象的未来观察和子任务描述逐步引导VLA。通过想象的未来观察,VLA可以专注于视觉-运动推理,而不是高级语义推理,从而提高准确性和泛化能力。我们的规划器包含一个高度高效的前瞻性图像生成模块,该模块能够在H100 GPU上仅用0.33秒从当前视觉输入和语言指令中预测出高质量的640×480的未来观察,同时还有一个视觉-语言模型,该模型对任务进行推理并为生成器和VLA生成子任务描述。重要的是,最先进的VLA可以无缝集成我们的规划器,只需增强其视觉输入,而无需任何架构修改。前瞻性生成器在超过100万个多任务、跨体感片段上进行预训练,使其能够学习稳健的体感动态。我们在一个包含11种多样、多步骤现实任务的基准上评估了我们的框架。它实现了87.4%的平均成功率,证明了相对于π₀基线(46.5%)的+40.9%的绝对提升,以及相对于π₀加上文本子任务指导(57.1%)的+30.3%的绝对提升。

英文摘要:

Vision-Language-Action (VLA) models convert high-level language instructions into concrete, executable actions, a task that is especially challenging in open-world environments. We present Visual Foresight Planning (ForeAct), a general and efficient planner that guides a VLA step-by-step using imagined future observations and subtask descriptions. With an imagined future observation, the VLA can focus on visuo-motor inference rather than high-level semantic reasoning, leading to improved accuracy and generalization. Our planner comprises a highly efficient foresight image generation module that predicts a high-quality 640$\times$480 future observation from the current visual input and language instruction within only 0.33s on an H100 GPU, together with a vision-language model that reasons over the task and produces subtask descriptions for both the generator and the VLA. Importantly, state-of-the-art VLAs can integrate our planner seamlessly by simply augmenting their visual inputs, without any architectural modification. The foresight generator is pretrained on over 1 million multi-task, cross-embodiment episodes, enabling it to learn robust embodied dynamics. We evaluate our framework on a benchmark that consists of 11 diverse, multi-step real-world tasks. It achieves an average success rate of 87.4%, demonstrating a +40.9% absolute improvement over the $π_0$ baseline (46.5%) and a +30.3% absolute improvement over $π_0$ augmented with textual subtask guidance (57.1%).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑