arXiv 2311.083842023-11-14cs.LGcs.AIstat.ML

离线数据增强的在线策略梯度方法及其可证明保证

Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees

Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；MIT（麻省理工学院）；Cornell University（康奈尔大学）

Yifei Zhou, Ayush Sekhari, Yuda Song, Wen Sun

备注：: The first two authors contributed equally

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出一种结合在线策略NPG与离线数据训练的混合强化学习算法，理论上实现离线与在线策略保证的兼得，实验上在丰富观测环境中优于仅用离策略优化的基线。

摘要：

混合强化学习（Hybrid RL）是智能体既能访问离线数据，又能通过与真实环境交互获取在线数据的设置。本文提出了一种新的混合强化学习算法，将在线策略的演员-评论家方法与离线数据相结合。策略梯度（Policy Gradient）和自然策略梯度（NPG）等在线策略方法在模型误设定下表现出更强的鲁棒性，但有时其样本效率可能不如依赖离策略学习的方法。另一方面，依赖离策略训练的离线方法在理论上通常需要较强的假设，且在实践中训练稳定性较差。我们的新方法将离线数据上的离策略训练过程整合到在线策略的NPG框架中。理论上，我们证明该方法能够获得“两全其美”的结果——当离线强化学习特有的假设成立时，它达到离线强化学习的最优理论保证；同时，无论这些假设是否成立，它都能保持在线策略NPG的理论保证。在实验上，面对具有挑战性的丰富观测环境，我们展示了该方法优于仅依赖离策略优化的最先进的混合强化学习基线，证明了结合在线与离策略学习的实证优势。我们的代码公开于 https://github.com/YifeiZhou02/HNPG。

英文摘要：

Hybrid RL is the setting where an RL agent has access to both offline data and online data by interacting with the real-world environment. In this work, we propose a new hybrid RL algorithm that combines an on-policy actor-critic method with offline data. On-policy methods such as policy gradient and natural policy gradient (NPG) have shown to be more robust to model misspecification, though sometimes it may not be as sample efficient as methods that rely on off-policy learning. On the other hand, offline methods that depend on off-policy training often require strong assumptions in theory and are less stable to train in practice. Our new approach integrates a procedure of off-policy training on the offline data into an on-policy NPG framework. We show that our approach, in theory, can obtain a best-of-both-worlds type of result -- it achieves the state-of-art theoretical guarantees of offline RL when offline RL-specific assumptions hold, while at the same time maintaining the theoretical guarantees of on-policy NPG regardless of the offline RL assumptions' validity. Experimentally, in challenging rich-observation environments, we show that our approach outperforms a state-of-the-art hybrid RL baseline which only relies on off-policy policy optimization, demonstrating the empirical benefit of combining on-policy and off-policy learning. Our code is publicly available at https://github.com/YifeiZhou02/HNPG.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。