arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.25477cs.ROcs.AI
收藏

EXPO-FT:面向视觉-语言-动作模型的样本高效强化学习微调

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

Stanford University(斯坦福大学)

Perry Dong, Kuo-Han Hung, Tian Gao, Dorsa Sadigh, Chelsea Finn

AI总结:

提出EXPO-FT系统,通过样本高效的强化学习微调预训练的VLA策略,在多种高精度操作任务中实现完美性能(30/30成功率),平均仅需19.1分钟在线机器人数据。

摘要:

高效且可靠地学习新任务的能力一直是机器人学的基础挑战。视觉-语言-动作(VLA)模型在多种操作任务中展现出强大的泛化能力,但预训练策略始终无法达到实际部署所需的可靠性。强化学习(RL)微调为弥合这一差距提供了有前景的路径,但现有方法要么从头开始训练而未充分利用预训练先验,要么微调VLA而未达到实际部署所需的样本效率和成功率。我们提出了EXPO-FT,一个用于对预训练VLA策略进行稳定、样本高效的RL微调的系统,填补了这一空白。我们的系统解决了一系列具有挑战性的操作任务,包括串灯并插入插头点亮、将台球击入袋中、将花插入酒瓶,每个任务都需要高精度、动态动作以及对不同初始状态的鲁棒性。我们的系统在所有评估任务中均实现了完美的任务性能(30/30成功),平均仅需19.1分钟的在线机器人数据,优于先前的从头RL训练和VLA微调方法。我们发布了一个开源代码库,旨在促进机器人领域中VLA模型RL微调的更广泛采用。

英文摘要:

The ability to efficiently and reliably learn new tasks has been a foundational challenge in robotics. Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse manipulation tasks, yet pretrained policies consistently fall short of the reliability required for real-world deployment. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches either train from scratch without fully leveraging pretrained priors, or fine-tune VLAs without achieving the sample efficiency and success rates that practical deployment demands. We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies that closes this gap. Our system solves a suite of challenging manipulation tasks, including routing string lights and inserting the plug to light it up, striking a pool ball into a pocket, and inserting a flower into a wine bottle, each requiring combinations of high precision, dynamic actions, and robustness to varied initial states. Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data, outperforming both prior RL-from-scratch and VLA finetuning approaches. We release an open-source codebase with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑