arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1811.03035cs.AI
收藏

计算规划的计算价值

Computing the Value of Computation for Planning

University College London(伦敦大学学院)

Can Eren Sezener

摘要:

智能代理执行操作以实现其目标。这样的动作可以是外部定向的,例如开门,或者是内部定向的,例如将数据写入存储器位置或加强突触连接。一些内部动作,我们称之为计算,可能有助于代理选择更好的动作。考虑到(外部)动作和计算可能会利用相同的资源,例如时间和能量,决定何时采取行动或计算,以及计算什么,对智能体的性能有害。在根据代理的行为提供奖励的环境中,动作的价值通常被定义为动作之后的预期长期奖励之和(本身是一个复杂的量,取决于代理在所讨论的动作之后继续做什么)。然而,定义计算的价值并不那么简单,因为计算只有在更高阶的方式下才有价值,通过改变动作。本论文提供了一个原则性的方法来计算的价值,在规划设置形式化为马尔可夫决策过程。我们提出了两种不同的计算值的定义:静态和动态。它们解决了计算预算的两种极端情况:提供未来零步或无限多步的计算。我们证明了这些值具有理想的性质,如时间一致性和渐近收敛性。此外,我们提出了有效地计算和近似的静态和动态计算值的方法。我们描述了一种意义上的政策,greetly最大化这些值可以是最优的。我们利用这些原则来构建蒙特卡罗树搜索算法,在相同的模拟资源下,在寻找更高质量的动作方面优于大多数最先进的算法。

英文摘要:

An intelligent agent performs actions in order to achieve its goals. Such actions can either be externally directed, such as opening a door, or internally directed, such as writing data to a memory location or strengthening a synaptic connection. Some internal actions, to which we refer as computations, potentially help the agent choose better actions. Considering that (external) actions and computations might draw upon the same resources, such as time and energy, deciding when to act or compute, as well as what to compute, are detrimental to the performance of an agent. In an environment that provides rewards depending on an agent's behavior, an action's value is typically defined as the sum of expected long-term rewards succeeding the action (itself a complex quantity that depends on what the agent goes on to do after the action in question). However, defining the value of a computation is not as straightforward, as computations are only valuable in a higher order way, through the alteration of actions. This thesis offers a principled way of computing the value of a computation in a planning setting formalized as a Markov decision process. We present two different definitions of computation values: static and dynamic. They address two extreme cases of the computation budget: affording calculation of zero or infinitely many steps in the future. We show that these values have desirable properties, such as temporal consistency and asymptotic convergence. Furthermore, we propose methods for efficiently computing and approximating the static and dynamic computation values. We describe a sense in which the policies that greedily maximize these values can be optimal. We utilize these principles to construct Monte Carlo tree search algorithms that outperform most of the state-of-the-art in terms of finding higher quality actions given the same simulation resources.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑