行动者-顾问：具有政策外建议的策略梯度
The Actor-Advisor: Policy Gradient With Off-Policy Advice
摘要：
Actor-critic算法学习一个显式策略（actor）和一个伴随的价值函数（critic）。参与者在环境中执行操作，而批评者评估参与者的当前策略。然而，尽管他们的稳定性和有前途的收敛性能，目前的演员批评家算法不优于批评，只有在实践中。我们认为，评论家学习Q^pi而不是最佳Q函数Q* 的事实阻止了使用最先进的鲁棒和样本高效的离策略学习算法。在本文中，我们提出了一个优雅的解决方案，演员顾问架构，其中一个策略梯度演员学习无偏蒙特卡罗返回，而被塑造（或建议）的Softmax政策所产生的政策外的批评。批评者可以独立于演员学习，使用任何最先进的算法。在高质量评论家的建议下，演员迅速而有力地学习任务，而其使用蒙特卡洛回归有助于克服评论家可能存在的任何偏见。除了新的Actor-Critic公式之外，Actor-Advisor（一种允许外部咨询策略塑造策略梯度参与者的方法）还可以应用于许多其他领域。通过改变建议的来源，我们证明了演员顾问的广泛适用性，其他三个重要的子领域的强化学习：安全的强化学习与备份策略，有效利用领域知识，并在强化学习迁移学习。我们的实验结果证明了Actor-Advisor与最先进的Actor-Critic方法相比的好处，说明了它对上面列出的其他三个应用场景的适用性，并表明RL的许多重要挑战现在可以使用一个优雅的解决方案来解决。
英文摘要：
Actor-critic algorithms learn an explicit policy (actor), and an accompanying value function (critic). The actor performs actions in the environment, while the critic evaluates the actor's current policy. However, despite their stability and promising convergence properties, current actor-critic algorithms do not outperform critic-only ones in practice. We believe that the fact that the critic learns Q^pi, instead of the optimal Q-function Q*, prevents state-of-the-art robust and sample-efficient off-policy learning algorithms from being used. In this paper, we propose an elegant solution, the Actor-Advisor architecture, in which a Policy Gradient actor learns from unbiased Monte-Carlo returns, while being shaped (or advised) by the Softmax policy arising from an off-policy critic. The critic can be learned independently from the actor, using any state-of-the-art algorithm. Being advised by a high-quality critic, the actor quickly and robustly learns the task, while its use of the Monte-Carlo return helps overcome any bias the critic may have. In addition to a new Actor-Critic formulation, the Actor-Advisor, a method that allows an external advisory policy to shape a Policy Gradient actor, can be applied to many other domains. By varying the source of advice, we demonstrate the wide applicability of the Actor-Advisor to three other important subfields of RL: safe RL with backup policies, efficient leverage of domain knowledge, and transfer learning in RL. Our experimental results demonstrate the benefits of the Actor-Advisor compared to state-of-the-art actor-critic methods, illustrate its applicability to the three other application scenarios listed above, and show that many important challenges of RL can now be solved using a single elegant solution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
