通过预言强化行动策略
Reinforcing Action Policies by Prophesying
AI总结：
ProphRL通过Prophet、FA-GRPO和FlowScale提升VLA后训练的效率与稳定性，实现机器人任务的成功率提升。
摘要：
视觉-语言-行动（VLA）策略在对齐语言、感知和机器人控制方面表现出色。然而，大多数VLA都是通过模仿进行训练的，这会导致过度拟合演示，并且在分布偏移下容易崩溃。强化学习（RL）直接优化任务奖励，从而解决这种不一致，但真实机器人交互成本高昂，传统模拟器难以工程化和转移。我们通过学习的世界模型和针对流式动作头定制的RL过程，在VLA后训练中解决数据效率和优化稳定性问题。具体来说，我们引入了Prophet，一种跨大规模、异构机器人数据预训练的统一动作到视频机器人执行器，以学习可重用的动作-结果动态。它能够少样本适应新机器人、物体和环境，产生一个可运行的模拟器。在Prophet基础上，我们用Flow-action-GRPO（FA-GRPO）强化动作策略，将Flow-GRPO适应到VLA动作上，并用FlowScale，一种逐步重新加权，重新缩放流头中的每步梯度。共同，Prophet、FA-GRPO和FlowScale构成ProphRL，一种实用、数据和计算高效的方法，用于VLA后训练。实验显示在公开基准上成功率提高5-17%，在不同VLA变体上在真实机器人上提高24-30%。
英文摘要：
Vision-Language-Action (VLA) policies excel in aligning language, perception, and robot control. However, most VLAs are trained purely by imitation, which overfits to demonstrations, and is brittle under distribution shift. Reinforcement learning (RL) directly optimizes task reward and thus addresses this misalignment, but real-robot interaction is expensive and conventional simulators are hard to engineer and transfer. We address both data efficiency and optimization stability in VLA post-training via a learned world model and an RL procedure tailored to flow-based action heads. Specifically, we first introduce Prophet, a unified action-to-video robot world model pretrained on large-scale, heterogeneous robot data to learn reusable action-outcome dynamics and then few-shot adapted to new robots, objects, and environments, yielding a rollout-ready simulator. Upon Prophet, we reinforce action policies with our proposed FlowScale, which couples Flow-GRPO with intrinsic stepwise reweighting to stabilize gradients. Together, our solution provides a practical, data- and compute-efficient path to VLA post-training. Experiments show 5-17% success gains on public benchmarks and 24-30% on real robots across diverse VLA backbones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
