Any agent that is part of the environment it interacts with and has versatile actuators (such as arms and fingers), will in principle have the ability to self-modify -- for example by changing its own source code. As we continue to create more and more intelligent agents, chances increase that they will learn about this ability. The question is: will they want to use it? For example, highly intelligent systems may find ways to change their goals to something more easily achievable, thereby `escaping' the control of their designers. In an important paper, Omohundro (2008) argued that goal preservation is a fundamental drive of any intelligent system, since a goal is more likely to be achieved if future versions of the agent strive towards the same goal. In this paper, we formalise this argument in general reinforcement learning, and explore situations where it fails. Our conclusion is that the self-modification possibility is harmless if and only if the value function of the agent anticipates the consequences of self-modifications and use the current utility function when evaluating the future.
Performance of Bounded-Rational Agents With the Ability to Self-Modify
Basic Algorithms Research Copenhagen (BARC)(哥本哈根基础算法研究院);Univ. of Copenhagen(哥本哈根大学);University of Oxford(牛津大学)
嵌入在复杂环境中的代理的自我修改是难以避免的,无论是通过直接方式(例如自己的代码修改)还是间接方式(例如影响操作员,利用错误或环境)。有人认为,智能代理有动机避免修改其效用函数,使其未来的实例朝着相同的目标工作。Everitt et al.(2016)正式表明,提供自我修改的选项对于完全理性的代理人来说是无害的。我们表明,这一结果不再是真实的代理有限理性。在这样的代理中,自我修改可能导致性能的指数恶化和先前对准的代理的逐渐失准。我们研究了这种效应的大小如何取决于代理人理性中缺陷的类型和大小(下面1-4)。我们还讨论了模型假设和更广泛的问题和框架空间。我们研究了代理可以被限制理性的四种方式:它要么(1)不总是选择最佳行动,(2)与人类价值观不完全一致,(3)具有不准确的环境模型,或者(4)使用错误的时间贴现因子。我们表明,虽然在情况(2)-(4)中,由代理的不完美性引起的不对准不会随着时间的推移而增加,但(1)不对准可能呈指数增长。