PIPPS：基于模型的灵活策略搜索
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos
摘要：
以前，爆炸梯度问题被解释为深度学习和基于模型的强化学习的核心，因为它会导致数值问题和优化的不稳定性。我们在基于模型的强化学习中的实验表明，这个问题不仅仅是一个数值问题，它可能是由长链非线性计算的基本混沌性质引起的。不仅梯度的幅度变大，梯度的方向也变得基本上随机。我们表明，重新参数化梯度遭受的问题，而似然比梯度是强大的。利用我们的见解，我们开发了一个基于模型的策略搜索框架，概率推理基于粒子的策略搜索（PIPPS），这是很容易扩展的，并允许几乎任意的模型和政策，同时匹配以前的数据高效的学习算法的性能。最后，我们发明了总传播算法，它有效地计算在一个单一的向后传递过程中的所有路径导数深度的联合，自动给予更大的权重，以较低的方差估计，有时改善了重新参数化梯度的$10^6$倍。
英文摘要：
Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may be caused by a fundamental chaos-like nature of long chains of nonlinear computations. Not only do the magnitudes of the gradients become large, the direction of the gradients becomes essentially random. We show that reparameterization gradients suffer from the problem, while likelihood ratio gradients are robust. Using our insights, we develop a model-based policy search framework, Probabilistic Inference for Particle-Based Policy Search (PIPPS), which is easily extensible, and allows for almost arbitrary models and policies, while simultaneously matching the performance of previous data-efficient learning algorithms. Finally, we invent the total propagation algorithm, which efficiently computes a union over all pathwise derivative depths during a single backwards pass, automatically giving greater weight to estimators with lower variance, sometimes improving over reparameterization gradients by $10^6$ times.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
