arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.12916cs.LG
收藏

一种策略有效的凸约束深度强化学习约简方法

A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning

Ant Group(蚂蚁集团)

Tianchi Cai, Wenpeng Zhang, Lihong Gu, Xiaodong Zeng, Jinjie Gu

摘要:

虽然在一般的强化学习(RL)中得到了很好的建立,但基于值的方法很少在约束RL(CRL)中进行探索,因为它们无法找到可以在多个动作中随机化的策略。为了将基于值的方法应用于CRL,最近一种开创性的博弈论方法使用了混合策略,该策略在一组精心生成的策略中随机化,以收敛到所需的满足约束的策略。然而,这些方法需要存储大量的策略,这不是策略有效的,并且可能在受约束的深度RL中引起过高的存储器成本。为了解决这个问题,我们提出了一种替代方法。我们的方法首先将CRL重新表述为一个等价的距离优化问题。通过一个专门设计的线性优化预言机,我们得到了一个元算法,该算法使用任何现成的RL算法和任何条件梯度(CG)类型的算法作为子程序来解决它。然后,我们提出了一个新的变体的CG型算法,推广的最小范数点(MNP)的方法。所提出的方法匹配现有的博弈论方法的收敛速度,并实现最坏情况下的最优策略效率。在导航任务上的实验表明,该方法在降低内存开销的同时,获得了更好的性能,证明了该方法的有效性和效率。

英文摘要:

Although well-established in general reinforcement learning (RL), value-based methods are rarely explored in constrained RL (CRL) for their incapability of finding policies that can randomize among multiple actions. To apply value-based methods to CRL, a recent groundbreaking line of game-theoretic approaches uses the mixed policy that randomizes among a set of carefully generated policies to converge to the desired constraint-satisfying policy. However, these approaches require storing a large set of policies, which is not policy efficient, and may incur prohibitive memory costs in constrained deep RL. To address this problem, we propose an alternative approach. Our approach first reformulates the CRL to an equivalent distance optimization problem. With a specially designed linear optimization oracle, we derive a meta-algorithm that solves it using any off-the-shelf RL algorithm and any conditional gradient (CG) type algorithm as subroutines. We then propose a new variant of the CG-type algorithm, which generalizes the minimum norm point (MNP) method. The proposed method matches the convergence rate of the existing game-theoretic approaches and achieves the worst-case optimal policy efficiency. The experiments on a navigation task show that our method reduces the memory costs by an order of magnitude, and meanwhile achieves better performance, demonstrating both its effectiveness and efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑