arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10503cs.RO
收藏

迈向长寿命机器人:通过强化微调实现持续学习的VLA模型

Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning

CASIA(中国科学院自动化研究所)

Yuan Liu, Haoran Li, Shuai Tian, Yuxing Qin, Yuhui Chen, Yupeng Zheng, Yongzhen Huang, Dongbin Zhao

AI总结:

本文提出LifeLong-RFT方法,通过整合分块级在线强化学习与多维过程奖励机制,提升VLA模型在多任务学习中的性能,实现持续学习。

摘要:

在大规模和多样化数据集上预训练的VLA模型表现出强大的泛化和适应能力作为通用机器人策略。然而,监督微调(SFT)作为适应VLA到下游领域的主要机制,需要大量任务特定数据且易发生灾难性遗忘。为解决这些限制,我们提出LifeLong-RFT,一种简单有效的强化微调(RFT)策略,独立于在线环境反馈和预训练奖励模型。通过整合分块级在线强化学习与所提出的多维过程奖励机制,LifeLong-RFT量化了中间动作分块在三个维度上的异质贡献,以促进策略优化。具体而言,(1)量化动作一致性奖励(QACR)确保在离散动作空间内的准确动作预测;(2)连续轨迹对齐奖励(CTAR)将解码的连续动作分块与参考轨迹对齐,以确保精确控制;(3)格式合规性奖励(FCR)保证输出的结构有效性。在SimplerEnv、LIBERO和现实任务中的全面实验表明,LifeLong-RFT在多任务学习中表现出色。此外,在LIBERO基准上的持续学习中,我们的方法在SFT基础上实现了22%的平均成功率提升,同时仅使用20%的训练数据即可有效适应新任务。整体而言,我们的方法提供了一种有前景的VLA后训练范式。项目页面可在<https://yuan-liu-lifelong-rft.github.io>获取。

英文摘要:

Pretrained on large-scale and diverse datasets, VLA models demonstrate strong generalization and adaptability as general-purpose robotic policies. However, Supervised Fine-Tuning (SFT), which serves as the primary mechanism for adapting VLAs to downstream domains, requires substantial amounts of task-specific data and is prone to catastrophic forgetting. To address these limitations, we propose LifeLong-RFT, a simple yet effective Reinforcement Fine-Tuning (RFT) strategy for VLA models independent of online environmental feedback and pre-trained reward models. By integrating chunking-level on-policy reinforcement learning with the proposed multi-dimensional process reward mechanism, LifeLong-RFT quantifies the heterogeneous contributions of intermediate action chunks across three dimensions to facilitate policy optimization. Specifically, (1) the Quantized Action Consistency Reward (QACR) ensures accurate action prediction within the discrete action space; (2) the Continuous Trajectory Alignment Reward (CTAR) aligns decoded continuous action chunks with reference trajectories to ensure precise control; (3) the Format Compliance Reward (FCR) guarantees the structural validity of outputs. Comprehensive experiments across SimplerEnv, LIBERO, and real-world tasks demonstrate that LifeLong-RFT exhibits strong performance in multi-task learning. Furthermore, for continual learning on the LIBERO benchmark, our method achieves a 22% gain in average success rate over SFT, while effectively adapting to new tasks using only 20% of the training data. Overall, our method provides a promising post-training paradigm for VLAs. The project page is available at <https://yuan-liu-lifelong-rft.github.io>.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑