通过隐式模仿引导的强化学习
Reinforcement Learning via Implicit Imitation Guidance
AI总结：
针对样本高效强化学习中模仿目标可能损害长期回报的问题，本文提出DGN框架，仅用演示数据通过策略噪声隐式引导探索，在七个连续控制任务上较离线数据RL方法最高提升2至3倍。
摘要：
我们研究样本高效强化学习问题，其中演示等先验数据被用于初始化，以替代稠密奖励信号。一种自然方法是纳入模仿学习目标，既可作为训练期间的正则化，也可用于获得参考策略。然而，模仿学习目标最终可能降低长期性能，因为它并不直接与奖励最大化对齐。在本研究中，我们提出仅将先验数据用于通过添加到策略中的噪声来引导探索，从而规避显式行为克隆约束的需要。我们的框架 Data-Guided Noise（DGN，数据引导噪声）的关键洞见在于，演示最有用之处是识别哪些动作应当被探索，而不是强迫策略采取某些动作。在七个模拟连续控制任务上，与以往基于离线数据的强化学习方法相比，我们的方法实现了最高2至3倍的提升。
英文摘要：
We study the problem of sample efficient reinforcement learning, where prior data such as demonstrations are provided for initialization in lieu of a dense reward signal. A natural approach is to incorporate an imitation learning objective, either as regularization during training or to acquire a reference policy. However, imitation learning objectives can ultimately degrade long-term performance, as it does not directly align with reward maximization. In this work, we propose to use prior data solely for guiding exploration via noise added to the policy, sidestepping the need for explicit behavior cloning constraints. The key insight in our framework, Data-Guided Noise (DGN), is that demonstrations are most useful for identifying which actions should be explored, rather than forcing the policy to take certain actions. Our approach achieves up to 2-3x improvement over prior reinforcement learning from offline data methods across seven simulated continuous control tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
