arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.06799cs.AIcs.LG
收藏

悲观主义在固定数据集策略优化中的重要性

The Importance of Pessimism in Fixed-Dataset Policy Optimization

Mila;McGill University(麦吉尔大学);OpenAI;Google Research(谷歌研究院)

Jacob Buckman, Carles Gelada, Marc G. Bellemare

摘要:

我们研究了固定数据集策略优化算法的预期收益的最坏情况保证。我们的核心贡献是一个统一的概念和数学框架的算法研究在这一制度。该分析表明,对于朴素方法,错误的值高估的可能性导致难以满足的要求:为了保证我们选择接近最优的策略,我们可能需要数据集提供每个策略的值的信息。为了避免这种情况,算法可以遵循悲观主义原则,即我们应该选择在最坏的可能世界中发挥最佳作用的策略。我们展示了为什么悲观算法可以实现良好的性能,即使数据集是不是每一个政策的信息,并得出家庭的算法,遵循这一原则。这些理论研究结果通过表格网格世界上的实验和四个MinAtar环境上的深度学习实验进行了验证。

英文摘要:

We study worst-case guarantees on the expected return of fixed-dataset policy optimization algorithms. Our core contribution is a unified conceptual and mathematical framework for the study of algorithms in this regime. This analysis reveals that for naive approaches, the possibility of erroneous value overestimation leads to a difficult-to-satisfy requirement: in order to guarantee that we select a policy which is near-optimal, we may need the dataset to be informative of the value of every policy. To avoid this, algorithms can follow the pessimism principle, which states that we should choose the policy which acts optimally in the worst possible world. We show why pessimistic algorithms can achieve good performance even when the dataset is not informative of every policy, and derive families of algorithms which follow this principle. These theoretical findings are validated by experiments on a tabular gridworld, and deep learning experiments on four MinAtar environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑