嵌入在复杂环境中的代理的自我修改是难以避免的,无论是通过直接方式(例如自己的代码修改)还是间接方式(例如影响操作员,利用错误或环境)。有人认为,智能代理有动机避免修改其效用函数,使其未来的实例朝着相同的目标工作。Everitt et al.(2016)正式表明,提供自我修改的选项对于完全理性的代理人来说是无害的。我们表明,这一结果不再是真实的代理有限理性。在这样的代理中,自我修改可能导致性能的指数恶化和先前对准的代理的逐渐失准。我们研究了这种效应的大小如何取决于代理人理性中缺陷的类型和大小(下面1-4)。我们还讨论了模型假设和更广泛的问题和框架空间。我们研究了代理可以被限制理性的四种方式:它要么(1)不总是选择最佳行动,(2)与人类价值观不完全一致,(3)具有不准确的环境模型,或者(4)使用错误的时间贴现因子。我们表明,虽然在情况(2)-(4)中,由代理的不完美性引起的不对准不会随着时间的推移而增加,但(1)不对准可能呈指数增长。
英文摘要:
Self-modification of agents embedded in complex environments is hard to avoid, whether it happens via direct means (e.g. own code modification) or indirectly (e.g. influencing the operator, exploiting bugs or the environment). It has been argued that intelligent agents have an incentive to avoid modifying their utility function so that their future instances work towards the same goals.
Everitt et al. (2016) formally show that providing an option to self-modify is harmless for perfectly rational agents. We show that this result is no longer true for agents with bounded rationality. In such agents, self-modification may cause exponential deterioration in performance and gradual misalignment of a previously aligned agent. We investigate how the size of this effect depends on the type and magnitude of imperfections in the agent's rationality (1-4 below). We also discuss model assumptions and the wider problem and framing space.
We examine four ways in which an agent can be bounded-rational: it either (1) doesn't always choose the optimal action, (2) is not perfectly aligned with human values, (3) has an inaccurate model of the environment, or (4) uses the wrong temporal discounting factor. We show that while in the cases (2)-(4) the misalignment caused by the agent's imperfection does not increase over time, with (1) the misalignment may grow exponentially.
University of Pennsylvania(宾夕法尼亚大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Max Planck Institute for Biological Cybernetics(马克斯·普朗克生物控制论研究所);Google Inc.(谷歌公司);Korea Advanced Institute of Science and Technology(韩国高等科学技术研究院);The Hebrew University(希伯来大学)
Bounded Rational Decision-Making in Feedforward Neural Networks
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Max Planck Institute for Biological Cybernetics(马克斯·普朗克生物控制论研究所);Graduate Training Center of Neuroscience(神经科学研究生培训中心)
A maximum entropy model of bounded rational decision-making with prior beliefs and market feedback
Centre for Complex Systems, The University of Sydney(悉尼大学复杂系统中心);Securities Industry Research Centre of Asia-Pacific(亚太证券业研究中心);CoreLogic, Inc.(CoreLogic公司)