ACCRUE：面向机器人基础模型的持续离线策略强化学习
ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
AI总结：
ACCRUE是一种持续离线策略强化学习框架，可跨任务复用轻量级演员与评论家网络，在近零遗忘的前提下加速后续任务学习，将真实机器人训练时间降低31.0%至68.5%。
摘要：
强化学习（RL）的后训练可提升机器人基础模型（RFM）的性能，使其超越演示水平，但实际部署还要求：高效学习新任务的同时保留已习得的原有行为。现有基于视觉语言动作模型（VLA）的持续RL方法虽能缓解遗忘，但依赖在线策略更新，正向迁移能力有限；而离线策略方法虽能提升单任务样本效率，却未解决跨任务的学习者持续性问题。本文提出ACCRUE，一种持续离线策略RL框架，该框架在跨任务时携带轻量级演员（actor）和评论家（critic）网络，使每个阶段可复用先前RL习得的行为与价值知识。继承的评论家仅在当前任务的转换数据上适配，并指导当前任务的演员学习；同时通过将演员蒸馏至少量保留片段的缓存目标，来保留先前行为。固定视觉特征使继承的学习者及其经验回放与基础策略更新兼容。在仿真及真实机器人硬件上的接触密集型装配序列实验中，ACCRUE可加速后续任务学习，遗忘率接近零，且将真实机器人训练时间显著降低31.0%至68.5%。
英文摘要：
Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0\% to 68.5\%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
