从分布式强化学习教师中持续政策蒸馏
Continual Policy Consolidation for Lifelong Robot Learning
AI总结：
本文提出一种教师-学生框架，通过分布式强化学习训练单任务模型并持续蒸馏到通用模型，实现高效的持续强化学习。
摘要：
持续强化学习（CRL）旨在开发能够持续学习的智能体，使其在多样任务中不断获取知识，同时缓解灾难性遗忘。这需要高效管理稳定性与可塑性之间的矛盾，并利用先前经验快速泛化到新任务。尽管已提出各种增强策略，但直接应用强化学习到连续任务流中仍具挑战性。本文提出了一种新的教师-学生框架，将CRL分解为两个独立过程：通过分布式强化学习训练单任务教师模型，并持续将其蒸馏到一个中央通用模型中。该设计受观察启发：强化学习擅长解决单任务，而政策蒸馏——一种相对稳定的监督学习过程——与大规模基础模型和多任务学习契合。此外，采用专家混合（MoE）架构和基于回放的方法来增强持续政策蒸馏过程的可塑性和稳定性。在Meta-World基准上的广泛实验表明，我们的框架实现了高效的持续强化学习，恢复超过85%的教师性能，同时将任务间的遗忘控制在10%以内。
英文摘要：
Building a generalist robot policy requires continuously integrating new skills while preserving previously acquired behaviors. Directly optimizing a single policy over a growing task stream is difficult because robotic interaction is expensive, task distributions are heterogeneous, and sequential updates induce interference. To address these problems, we propose continual policy consolidation (CPC), a teacher--student framework that combines continual policy distillation with prioritized experience replay and expandable experts. This architecture separates skill acquisition from policy consolidation: teachers are trained independently through reinforcement learning, and their behaviors are continually distilled into a central generalist student. This decomposition retains the practical strength of reinforcement learning for task-specialized training while casting student-side consolidation as a supervised policy-learning problem. To balance stability and plasticity as the task stream grows, the student combines an expandable Transformer-based mixture-of-experts architecture with prioritized trajectory replay. Extensive experiments show that the student recovers a large proportion of teacher performance while achieving near-zero forgetting. These results demonstrate a scalable route for consolidating independently acquired robot skills into a continually growing generalist policy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
