arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.06865stat.MLcs.LG
收藏

双曲贴现与多水平学习

Hyperbolic Discounting and Learning over Multiple Horizons

Google Brain(谷歌大脑);University of Montreal (Mila)(蒙特利尔大学(米拉));CIFAR Senior Fellow(加拿大高级研究院资深研究员)

William Fedus, Carles Gelada, Yoshua Bengio, Marc G. Bellemare, Hugo Larochelle

摘要:

强化学习(RL)通常将折扣因子定义为马尔可夫决策过程的一部分。贴现因子的价值未来的奖励指数计划,导致理论上的贝尔曼方程的收敛保证。然而,来自心理学、经济学和神经科学的证据表明,人类和动物都有双曲线时间偏好。在这项工作中,我们重新审视了RL中折扣的基本原理,并通过实现一个通过双曲线折扣起作用的RL代理来弥合这种脱节。我们证明了一种简单的方法近似双曲折扣函数,同时仍然使用RL中熟悉的时间差学习技术。此外,独立于双曲贴现,我们令人惊讶地发现,在多个时间范围内同时学习值函数是一种有效的辅助任务,通常比基于值的强大RL代理Rainbow有所改进。

英文摘要:

Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑