Both animals and artificial agents benefit from state representations that support rapid transfer of learning across tasks and which enable them to efficiently traverse their environments to reach rewarding states. The successor representation (SR), which measures the expected cumulative, discounted state occupancy under a fixed policy, enables efficient transfer to different reward structures in an otherwise constant Markovian environment and has been hypothesized to underlie aspects of biological behavior and neural activity. However, in the real world, rewards may move or only be available for consumption once, may shift location, or agents may simply aim to reach goal states as rapidly as possible without the constraint of artificially imposed task horizons. In such cases, the most behaviorally-relevant representation would carry information about when the agent was likely to first reach states of interest, rather than how often it should expect to visit them over a potentially infinite time span. To reflect such demands, we introduce the first-occupancy representation (FR), which measures the expected temporal discount to the first time a state is accessed. We demonstrate that the FR facilitates exploration, the selection of efficient paths to desired states, allows the agent, under certain conditions, to plan provably optimal trajectories defined by a sequence of subgoals, and induces similar behavior to animals avoiding threatening stimuli.
Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization
Computational Statistics and Machine Learning - Istituto Italiano di Tecnologia(计算统计与机器学习 - 意大利技术研究院);AI Centre, Computer Science Department, University College London(人工智能中心,计算机科学系,伦敦大学学院)
On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning
Singapore University of Technology and Design(新加坡科技设计大学);University of Maryland-College Park(马里兰大学-College Park分校);University of Central Florida(佛罗里达中央大学)
Avoidance Learning Using Observational Reinforcement Learning
McGill University(麦吉尔大学);Mila(米拉人工智能研究所);MIT(麻省理工学院);Université de Montréal(蒙特利尔大学);École CentraleSupélec(中央理工-高等电力学院);DeepMind(深度思维)
模仿学习试图从抽样演示中学习专家策略。然而,在现实世界中,通常很难找到完美的专家,出于安全原因,避免危险行为变得至关重要。我们提出了\textit{learning to avoid}的思想,在某种意义上,一个目标相反的模仿学习,其中代理学习,以避免演示政策给定的环境。我们将回避学习定义为优化智能体奖励同时避免演示者给出的危险行为的过程。在这项工作中,我们开发了一个框架,通过定义一个合适的目标函数,这些问题涉及的专家和演示政策的状态占用分布的\emph{distance}的避免学习。我们使用密度估计的状态占用措施,并使用上述距离作为奖励奖金,以避免示威者。我们验证我们的理论与实验使用范围广泛的部分可观察的环境。实验结果表明,我们能够提高样本效率在训练过程中相比,最先进的政策优化和安全的方法。
Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilities
ETH Zurich, Zurich, Switzerland(苏黎世联邦理工学院);Max Planck Institute for Intelligent Systems, Tubingen, Germany(智能系统马克斯·普朗克研究所);University of Tubingen, Tubingen, Germany(图宾根大学)
Auditing Near-Optimal Policies Can Be Exponentially Hard: Conditional Query Lower Bounds via Occupancy Rashomon Capacity
Department of Computer Science, Iowa State University(计算机科学系,爱荷华州立大学);Department of Civil, Construction & Environmental Engineering, Iowa State University(土木、建设与环境工程系,爱荷华州立大学)