视觉-语言-动作模型的交互式后训练
Interactive Post-Training for Vision-Language-Action Models
AI总结：
提出RIPT-VLA，一种基于强化学习的交互式后训练方法，仅用稀疏二元奖励微调VLA模型，显著提升性能并实现低数据下的高效泛化。
摘要：
我们引入了RIPT-VLA，一种简单且可扩展的基于强化学习的交互式后训练范式，它仅使用稀疏的二元成功奖励来微调预训练的视觉-语言-动作（VLA）模型。现有的VLA训练流程严重依赖离线专家演示数据和监督式模仿，这限制了它们在低数据场景下适应新任务和新环境的能力。RIPT-VLA通过采用基于动态回滚采样和留一法优势估计的稳定策略优化算法，实现了交互式后训练，从而解决了这一问题。RIPT-VLA具有以下特点。首先，它适用于各种VLA模型，使轻量级QueST模型提升了21.2%，并使7B OpenVLA-OFT模型达到了前所未有的97.5%成功率。其次，它在计算和数据上高效：仅凭一个演示，RIPT-VLA就能在15次迭代内使一个不可用的SFT模型（4%）达到97%的成功率。此外，我们证明了RIPT-VLA学习到的策略能够跨不同任务和场景泛化，并且对初始状态上下文具有鲁棒性。这些结果凸显了RIPT-VLA作为一种通过最小监督对VLA模型进行后训练的实用且有效的范式。
英文摘要：
We introduce RIPT-VLA, a simple and scalable reinforcement-learning-based interactive post-training paradigm that fine-tunes pretrained Vision-Language-Action (VLA) models using only sparse binary success rewards. Existing VLA training pipelines rely heavily on offline expert demonstration data and supervised imitation, limiting their ability to adapt to new tasks and environments under low-data regimes. RIPT-VLA addresses this by enabling interactive post-training with a stable policy optimization algorithm based on dynamic rollout sampling and leave-one-out advantage estimation. RIPT-VLA has the following characteristics. First, it applies to various VLA models, resulting in an improvement on the lightweight QueST model by 21.2%, and the 7B OpenVLA-OFT model to an unprecedented 97.5% success rate. Second, it is computationally efficient and data-efficient: with only one demonstration, RIPT-VLA enables an unworkable SFT model (4%) to succeed with a 97% success rate within 15 iterations. Furthermore, we demonstrate that the policy learned by RIPT-VLA generalizes across different tasks and scenarios and is robust to the initial state context. These results highlight RIPT-VLA as a practical and effective paradigm for post-training VLA models through minimal supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
