arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.13863cs.LGcs.AI
收藏

一种用于强化学习的第一占用表示

A First-Occupancy Representation for Reinforcement Learning

UCL(伦敦大学学院)

Ted Moskovitz, Spencer R. Wilson, Maneesh Sahani

摘要:

动物和人工智能体都受益于状态表示,这些状态表示支持跨任务的快速学习转移,并使它们能够有效地穿越环境以达到奖励状态。后继表示(SR),它测量预期的累积,折扣状态占用下一个固定的政策,使有效的转移到不同的奖励结构,否则恒定的马尔可夫环境,并已被假设为生物行为和神经活动的基础方面。然而,在现实世界中,奖励可能会移动或仅可用于消费一次,可能会改变位置,或者代理可能只是旨在尽可能快地达到目标状态,而不受人为强加的任务范围的约束。在这种情况下,行为上最相关的表示将携带关于智能体何时可能首次到达感兴趣状态的信息,而不是在潜在的无限时间跨度内应该多久访问一次。为了反映这种需求,我们引入了第一占用表示(FR),它测量了第一次访问状态时的预期时间折扣。我们表明,FR促进探索,选择有效的路径到所需的状态,允许代理,在某些条件下,计划可证明的最佳轨迹定义的一系列子目标,并诱导类似的行为,动物避免威胁刺激。

英文摘要:

Both animals and artificial agents benefit from state representations that support rapid transfer of learning across tasks and which enable them to efficiently traverse their environments to reach rewarding states. The successor representation (SR), which measures the expected cumulative, discounted state occupancy under a fixed policy, enables efficient transfer to different reward structures in an otherwise constant Markovian environment and has been hypothesized to underlie aspects of biological behavior and neural activity. However, in the real world, rewards may move or only be available for consumption once, may shift location, or agents may simply aim to reach goal states as rapidly as possible without the constraint of artificially imposed task horizons. In such cases, the most behaviorally-relevant representation would carry information about when the agent was likely to first reach states of interest, rather than how often it should expect to visit them over a potentially infinite time span. To reflect such demands, we introduce the first-occupancy representation (FR), which measures the expected temporal discount to the first time a state is accessed. We demonstrate that the FR facilitates exploration, the selection of efficient paths to desired states, allows the agent, under certain conditions, to plan provably optimal trajectories defined by a sequence of subgoals, and induces similar behavior to animals avoiding threatening stimuli.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑