arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.10316cs.AIcs.LGq-bio.NC
收藏

源于占据动作-状态路径空间的内在动机所产生的复杂行为

Complex behavior from intrinsic motivation to occupy action-state path space

Center for Brain and Cognition(大脑与认知中心);Universitat Pompeu Fabra(庞培法布拉大学)

Jorge Ramírez-Ruiz, Dmytro Grytskyy, Chiara Mastrogiuseppe, Yamen Habib, Rubén Moreno-Bote

AI总结:

本文提出最大占据原则,以最大化未来动作-状态路径占据取代奖励最大化,证明路径熵是唯一一致度量,并展示该内在动机可自然涌现跳舞、捉迷藏和利他等复杂行为。

摘要:

大多数行为理论假定智能体倾向于最大化某种形式的奖励或效用。然而,动物常常带着好奇心移动,似乎以一种无奖励的方式受到激励。在此,我们放弃奖励最大化的思想,并提出行为的目标是最大化未来动作和状态路径的占据。根据这一最大占据原则,奖励是占据路径空间的手段,而非目标本身;目标导向性只是作为寻找资源的理性方式而自然涌现,从而使运动(从广义上理解)永不终止。我们发现,动作-状态路径熵是唯一与预期未来动作-状态路径占据的可加性及其他直观性质相一致的度量。我们给出了联系最优策略与状态价值函数的解析表达式,并证明了我们的价值迭代算法的收敛性。通过使用离散和连续状态任务(包括一个高维控制器),我们表明,诸如“跳舞”、捉迷藏以及一种基本形式的利他行为等复杂行为,会自然地从占据路径空间的内在动机中产生。总而言之,我们提出了一种在缺乏奖励最大化的情况下同时产生变异性和目标导向性的行为理论。

英文摘要:

Most theories of behavior posit that agents tend to maximize some form of reward or utility. However, animals very often move with curiosity and seem to be motivated in a reward-free manner. Here we abandon the idea of reward maximization, and propose that the goal of behavior is maximizing occupancy of future paths of actions and states. According to this maximum occupancy principle, rewards are the means to occupy path space, not the goal per se; goal-directedness simply emerges as rational ways of searching for resources so that movement, understood amply, never ends. We find that action-state path entropy is the only measure consistent with additivity and other intuitive properties of expected future action-state path occupancy. We provide analytical expressions that relate the optimal policy and state-value function, and prove convergence of our value iteration algorithm. Using discrete and continuous state tasks, including a high--dimensional controller, we show that complex behaviors such as `dancing', hide-and-seek and a basic form of altruistic behavior naturally result from the intrinsic motivation to occupy path space. All in all, we present a theory of behavior that generates both variability and goal-directedness in the absence of reward maximization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑