ClipUp：一个简单而强大的分布式策略优化器
ClipUp: A Simple and Powerful Optimizer for Distribution-based Policy Evolution
摘要：
基于分布的搜索算法是神经网络控制器进化强化学习的一种有效方法。在这些算法中，总回报相对于策略参数的梯度使用从搜索分布中提取的解的种群来估计，然后用于具有随机梯度上升的策略优化。社区中的一个常见选择是使用Adam优化算法在梯度上升期间获得自适应行为，这是由于其在各种监督学习设置中的成功。作为Adam的替代方案，我们提出了两种简单的技术来增强经典的基于动量的梯度上升：梯度归一化和更新裁剪。我们认为，由此产生的优化器称为ClipUp（简称“剪辑更新”）是一个更好的选择，基于分布的政策演变，因为它的工作原理是简单，易于理解，其超参数可以更直观地调整在实践中。此外，如果奖励规模发生变化，它不需要重新调整超参数。实验表明，ClipUp尽管简单，但与Adam竞争，并且在具有挑战性的连续控制基准上是有效的，包括基于Bullet物理模拟器的Humanoid控制任务。
英文摘要：
Distribution-based search algorithms are an effective approach for evolutionary reinforcement learning of neural network controllers. In these algorithms, gradients of the total reward with respect to the policy parameters are estimated using a population of solutions drawn from a search distribution, and then used for policy optimization with stochastic gradient ascent. A common choice in the community is to use the Adam optimization algorithm for obtaining an adaptive behavior during gradient ascent, due to its success in a variety of supervised learning settings. As an alternative to Adam, we propose to enhance classical momentum-based gradient ascent with two simple techniques: gradient normalization and update clipping. We argue that the resulting optimizer called ClipUp (short for "clipped updates") is a better choice for distribution-based policy evolution because its working principles are simple and easy to understand and its hyperparameters can be tuned more intuitively in practice. Moreover, it removes the need to re-tune hyperparameters if the reward scale changes. Experiments show that ClipUp is competitive with Adam despite its simplicity and is effective on challenging continuous control benchmarks, including the Humanoid control task based on the Bullet physics simulator.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
