arXiv 2503.109492025-03-13cs.ROcs.AI

机器人强化学习策略 Sim2Real 迁移后的安全持续域适应

Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics

Technical University of Munich（慕尼黑工业大学）；University of California, Berkeley（加州大学伯克利分校）；Technische Universität Berlin（柏林工业大学）；L3S Research Center, Leibniz Universität Hannover（莱布尼茨汉诺威大学L3S研究中心）

Josip Josifovski, Shangding Gu, Mohammadhossein Malmir, Haoliang Huang, Sayantan Auddy, Nicolás Navarro-Guerrero, Costas Spanos, Alois Knoll

备注：: 8 pages, 5 figures, under review

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究结合安全强化学习与持续学习，使机器人策略在 sim2real 部署后能安全适应当前真实域分布和动力学，并避免灾难性遗忘。

摘要：

域随机化已成为强化学习（RL）中的一项基础技术，用于促进策略从仿真向真实机器人应用的迁移。现有许多域随机化方法被提出以提升鲁棒性和 sim2real 迁移效果。这些方法依赖较宽的随机化范围来补偿未知的真实系统参数，从而产生鲁棒但低效的真实世界策略。此外，由于基于 RL 的优化过程本身具有不稳定性，并且需要在真实系统上采样利用性但可能不安全的动作，在域随机化仿真中预训练的策略在部署后会被固定。这限制了已部署策略对随时间不可避免变化的系统参数或环境动力学的适应能力。我们利用安全 RL 和持续学习，并结合域随机化仿真来解决这些局限，从而在真实机器人控制中实现安全的部署时策略适应。实验表明，我们的方法使策略能够适应并拟合真实系统当前的域分布和环境动力学，同时最大限度降低安全风险，并避免预训练阶段在随机化仿真中得到的通用策略发生灾难性遗忘等问题。视频和补充材料可在 https://safe-cda.github.io/ 获取。

英文摘要：

Domain randomization has emerged as a fundamental technique in reinforcement learning (RL) to facilitate the transfer of policies from simulation to real-world robotic applications. Many existing domain randomization approaches have been proposed to improve robustness and sim2real transfer. These approaches rely on wide randomization ranges to compensate for the unknown actual system parameters, leading to robust but inefficient real-world policies. In addition, the policies pretrained in the domain-randomized simulation are fixed after deployment due to the inherent instability of the optimization processes based on RL and the necessity of sampling exploitative but potentially unsafe actions on the real system. This limits the adaptability of the deployed policy to the inevitably changing system parameters or environment dynamics over time. We leverage safe RL and continual learning under domain-randomized simulation to address these limitations and enable safe deployment-time policy adaptation in real-world robot control. The experiments show that our method enables the policy to adapt and fit to the current domain distribution and environment dynamics of the real system while minimizing safety risks and avoiding issues like catastrophic forgetting of the general policy found in randomized simulation during the pretraining phase. Videos and supplementary material are available at https://safe-cda.github.io/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。