arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.26006cs.ROcs.AI
收藏

FORCE: 通过值校准预热和自蒸馏实现高效的VLA强化学习微调

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Institute of Automation, Chinese Academy of Sciences, Beijing, China(中国科学院自动化研究所);Beijing Academy of Artificial Intelligence, Beijing, China(北京人工智能研究院);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China(北京大学计算机科学学院多媒体信息处理国家重点实验室)

Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang

AI总结:

提出FORCE框架,通过值校准预热和自蒸馏解决VLA模型RL微调中的样本效率低和初始遗忘问题,在仿真和真实任务中成功率提升79%,训练加速32.5%。

摘要:

视觉-语言-动作(VLA)模型通常受到次优数据导致的模仿上限的限制。虽然强化学习(RL)微调可以超越这一限制,但其样本效率极低。这一挑战源于两个核心问题:(1)由于不稳定的Q函数导致的灾难性初始遗忘,以及(2)低质量探索数据导致的低效策略更新,通常迫使依赖昂贵的人工干预。我们提出了FORCE,一个三阶段框架,通过解决这两个问题来稳定微调。FORCE首先引入值校准预热阶段,利用在线策略rollout来缓解Q函数的分布偏移。随后,在在线阶段,这个校准后的Q函数充当过滤器,用于策略自身的动作提议和专家数据,确保只有高价值动作被用于策略更新。我们在各种仿真和真实世界任务上评估了FORCE,结果显示FORCE在成功率上实现了79%的绝对提升,比先前的RL方法高出10%,同时训练加速32.5%。关键的是,它缓解了常见的成功率下降问题,并在无需人工干预的情况下实现了这一稳健性能,朝着部署有能力的自主机器人智能体迈出了重要一步。

英文摘要:

Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑