Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.
UGAE: A Novel Approach to Non-exponential Discounting
École Polytechnique(巴黎综合理工学院);CNRS(法国国家科学研究中心);Institut Polytechnique de Paris(巴黎理工学院);University of Rennes(雷恩大学);Inria(法国国家信息与自动化研究所);IRISA(法国雷恩信息系统与随机系统研究所)
Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Tasks
University of Toronto(多伦多大学);Vector Institute(向量研究所);Pontificia Universidad Católica de Chile(智利天主教大学);Centro Nacional de Inteligencia Artificial(国家人工智能中心);Schwartz Reisman Institute(施瓦茨·赖斯曼研究所)