残差离策略强化学习用于微调行为克隆策略
Residual Off-Policy RL for Finetuning Behavior Cloning Policies
AI总结：
提出残差学习框架，以行为克隆策略为黑盒基础，用离策略强化学习学习轻量残差修正，仅需稀疏二值奖励，首次在灵巧手人形机器人上实现真实世界RL训练，达到最先进性能。
摘要：
行为克隆（BC）的最新进展已实现了令人印象深刻的视觉运动控制策略。然而，这些方法受限于人类示范的质量、数据收集所需的手动工作以及离线数据收益递减的问题。相比之下，强化学习（RL）通过与环境自主交互来训练智能体，并在各个领域取得了显著成功。尽管如此，直接在真实机器人上训练RL策略仍然具有挑战性，原因在于样本效率低下、安全问题以及从稀疏奖励中学习长时程任务（尤其是高自由度（DoF）系统）的困难。我们提出了一种通过残差学习框架结合BC和RL优势的方案。我们的方法将BC策略作为黑盒基础，并通过样本高效的离策略RL学习轻量级的逐步骤残差修正。我们证明，我们的方法仅需稀疏的二值奖励信号，就能在仿真和现实世界中有效改进高自由度（DoF）系统的操作策略。特别是，我们展示了据我们所知首次在具有灵巧手的人形机器人上成功进行真实世界RL训练。我们的结果在各种基于视觉的任务中展示了最先进的性能，为在现实世界中部署RL指明了一条实用路径。
英文摘要：
Recent advances in behavior cloning (BC) have enabled impressive visuomotor control policies. However, these approaches are limited by the quality of human demonstrations, the manual effort required for data collection, and the diminishing returns from offline data. In comparison, reinforcement learning (RL) trains an agent through autonomous interaction with the environment and has shown remarkable success in various domains. Still, training RL policies directly on real-world robots remains challenging due to sample inefficiency, safety concerns, and the difficulty of learning from sparse rewards for long-horizon tasks, especially for high-degree-of-freedom (DoF) systems. We present a recipe that combines the benefits of BC and RL through a residual learning framework. Our approach leverages BC policies as black-box bases and learns lightweight per-step residual corrections via sample-efficient off-policy RL. We demonstrate that our method requires only sparse binary reward signals and can effectively improve manipulation policies on high-degree-of-freedom (DoF) systems in both simulation and the real world. In particular, we demonstrate, to the best of our knowledge, the first successful real-world RL training on a humanoid robot with dexterous hands. Our results demonstrate state-of-the-art performance in various vision-based tasks, pointing towards a practical pathway for deploying RL in the real world.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
