利用从离线数据中元学习的目标增强在线强化学习
Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
AI总结：
针对稀疏奖励下在线强化学习难学最优策略的问题，提出可适配多种离策略RL的GILD模块，从离线数据元学习目标以注入内在动机，在MuJoCo稀疏奖励任务上性能超越现有最优方法。
摘要：
强化学习（RL）的一大核心挑战是难以从稀疏奖励中学习到最优策略。现有研究通过手工设计的辅助目标，结合传统模仿学习（IL）来增强在线RL，但代价是当离线数据由非专家策略生成时，RL策略会受限于次优水平。为更好地利用离线数据中的宝贵信息，我们提出了通用示范模仿学习（Generalized Imitation Learning from Demonstration, GILD），该方法通过元学习得到一个目标，既能从离线数据中提炼知识，又能为最优策略注入内在动机。与仅适用于特定RL算法的现有方法不同，GILD是一个灵活的模块，可适配多种常规离策略RL算法。此外，GILD无需引入领域特定超参数，计算成本增加极小。在四项带稀疏奖励的高难度MuJoCo任务中，我们验证了经GILD增强的三种RL算法性能显著优于当前最优方法。
英文摘要：
A major challenge in Reinforcement Learning (RL) is the difficulty of learning an optimal policy from sparse rewards. Prior works enhance online RL with conventional Imitation Learning (IL) via a handcrafted auxiliary objective, at the cost of restricting the RL policy to be sub-optimal when the offline data is generated by a non-expert policy. Instead, to better leverage valuable information in offline data, we develop Generalized Imitation Learning from Demonstration (GILD), which meta-learns an objective that distills knowledge from offline data and instills intrinsic motivation towards the optimal policy. Distinct from prior works that are exclusive to a specific RL algorithm, GILD is a flexible module intended for diverse vanilla off-policy RL algorithms. In addition, GILD introduces no domain-specific hyperparameter and minimal increase in computational cost. In four challenging MuJoCo tasks with sparse rewards, we show that three RL algorithms enhanced with GILD significantly outperform state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
