分解的策略梯度：MOMDPs中有效学习的杠杆结构
Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs
摘要：
策略梯度方法可以解决复杂的任务，但往往失败时，行动空间的维数或目标的多样性变得非常大。这部分是因为基于分数的梯度估计器的方差是二次缩放的。在本文中，我们解决这个问题，通过一个因素基线，利用独立结构编码在一个新的行动目标的影响网络。因子化的策略梯度（FPGs），这如下，提供了一个共同的框架，用于分析关键国家的最先进的算法，概括传统的策略梯度，并产生一个原则性的方式，将先验知识的问题域的生成过程。我们提供了一个分析建议的估计，并确定在何种条件下，方差减少。FPGs的算法方面进行了讨论，包括最佳的政策因式分解，其特征在于最小biclique覆盖，以及不正确地指定网络的偏差方差权衡的影响。最后，我们证明了我们的算法在大规模的土匪和交通路口的问题上的性能优势，后者提供了一个新的贡献，在空间近似的形式。
英文摘要：
Policy gradient methods can solve complex tasks but often fail when the dimensionality of the action-space or objective multiplicity grow very large. This occurs, in part, because the variance on score-based gradient estimators scales quadratically. In this paper, we address this problem through a factor baseline which exploits independence structure encoded in a novel action-target influence network. Factored policy gradients (FPGs), which follow, provide a common framework for analysing key state-of-the-art algorithms, are shown to generalise traditional policy gradients, and yield a principled way of incorporating prior knowledge of a problem domain's generative processes. We provide an analysis of the proposed estimator and identify the conditions under which variance is reduced. The algorithmic aspects of FPGs are discussed, including optimal policy factorisation, as characterised by minimum biclique coverings, and the implications for the bias-variance trade-off of incorrectly specifying the network. Finally, we demonstrate the performance advantages of our algorithm on large-scale bandit and traffic intersection problems, providing a novel contribution to the latter in the form of a spatial approximation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
