arXiv 2401.001622024-08-03cs.LG

基于仅状态示范学习平滑引导的策略优化

Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations

School of Mathematical Sciences, Beihang University（北京航空航天大学数学科学学院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Zhongguancun Laboratory（中关村实验室）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（未来区块链与隐私计算北京高精尖创新中心）

Guojian Wang, Faguo Wu, Xiao Zhang, Tianyuan Chen

备注：: 31 pages, 23 figures; This work has been submitted to the IEEE for possible publication

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对稀疏奖励与高质量动作示范难获取问题，提出POSG算法，利用仅状态示范评估轨迹重要性并生成平滑引导奖励，理论给出性能提升下界，且在四类环境中验证其控制与收敛优势。

摘要：

奖励反馈的稀疏性仍是在线深度强化学习（DRL）中的一个挑战性问题。以往方法利用离线示范，在多个困难任务中取得了令人瞩目的结果。然而，这些方法对示范质量要求很高，而获得类似专家的动作通常成本高昂且不现实。为解决这些问题，我们提出了一种简单高效的算法，称为基于平滑引导的策略优化（POSG），它利用少量仅状态示范（示范中不包含专家动作信息）来间接进行近似且可行的长期信用分配，并促进探索。具体而言，我们首先设计了一种轨迹重要性评估机制，以对照示范判断当前轨迹的质量。随后，我们引入一种基于轨迹重要性的引导奖励计算技术，用于衡量每个状态-动作对的影响，并将示范者的状态分布与奖励信息融合到引导奖励中。我们从理论上分析了平滑引导奖励带来的性能提升，并推导了一个新的性能提升最坏情况下界。大量结果表明，在包括网格世界迷宫、Hopper-v4、HalfCheetah-v4和Ant迷宫在内的四种稀疏奖励环境中，POSG在控制性能和收敛速度方面具有显著优势。值得注意的是，本文研究了具体指标和可量化结果，以证明POSG的优越性。

英文摘要：

The sparsity of reward feedback remains a challenging problem in online deep reinforcement learning (DRL). Previous approaches have utilized offline demonstrations to achieve impressive results in multiple hard tasks. However, these approaches place high demands on demonstration quality, and obtaining expert-like actions is often costly and unrealistic. To tackle these problems, we propose a simple and efficient algorithm called Policy Optimization with Smooth Guidance (POSG), which leverages a small set of state-only demonstrations (where expert action information is not included in demonstrations) to indirectly make approximate and feasible long-term credit assignments and facilitate exploration. Specifically, we first design a trajectory-importance evaluation mechanism to determine the quality of the current trajectory against demonstrations. Then, we introduce a guidance reward computation technology based on trajectory importance to measure the impact of each state-action pair, fusing the demonstrator's state distribution with reward information into the guidance reward. We theoretically analyze the performance improvement caused by smooth guidance rewards and derive a new worst-case lower bound on the performance improvement. Extensive results demonstrate POSG's significant advantages in control performance and convergence speed in four sparse-reward environments, including the grid-world maze, Hopper-v4, HalfCheetah-v4, and Ant maze. Notably, the specific metrics and quantifiable results are investigated to demonstrate the superiority of POSG.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。