基于效用函数迭代改进的AGI Agent安全
AGI Agent Safety by Iteratively Improving the Utility Function
摘要：
虽然目前还不清楚是否可以构建具有人工通用智能（AGI）的代理，但我们已经可以使用数学模型来研究这些代理的潜在安全系统。我们提出了一个AGI安全层，创建一个特殊的专用输入终端，以支持AGI代理的效用函数的迭代改进。开启智能体的人可以使用这个终端来关闭在效用函数对智能体目标和约束的编码中发现的任何漏洞，将智能体引向新的目标，或者迫使智能体关闭自己。AGI代理可能会开发出操纵上述效用函数改进过程的紧急激励，例如通过欺骗，限制甚至攻击所涉及的人类。安全层将部分地、有时完全地抑制这种危险的激励。本文的第一部分概括了早期工作的AGI紧急停止按钮。我们的目标是使用于构建层的数学方法更容易访问，通过将它们应用到MDP模型。我们讨论了两个可证明的属性的安全层，并显示正在进行的工作，将其映射到一个因果影响图（CID）。在第二部分中，我们开发了完整的数学证明，并表明，安全层创建一种类型的官僚盲目性。然后，我们提出了一个学习代理的设计，一个设计，包装的安全层周围的一个已知的机器学习系统，或潜在的未来AGI级学习系统。由此产生的代理将满足可证明的安全属性，从那一刻起，它是第一次打开。最后，我们展示了这个代理可以从它的模型映射到现实生活中的实现。我们回顾了这一步骤中涉及的方法问题，并讨论了这些问题通常是如何解决的。
英文摘要：
While it is still unclear if agents with Artificial General Intelligence (AGI) could ever be built, we can already use mathematical models to investigate potential safety systems for these agents. We present an AGI safety layer that creates a special dedicated input terminal to support the iterative improvement of an AGI agent's utility function. The humans who switched on the agent can use this terminal to close any loopholes that are discovered in the utility function's encoding of agent goals and constraints, to direct the agent towards new goals, or to force the agent to switch itself off. An AGI agent may develop the emergent incentive to manipulate the above utility function improvement process, for example by deceiving, restraining, or even attacking the humans involved. The safety layer will partially, and sometimes fully, suppress this dangerous incentive. The first part of this paper generalizes earlier work on AGI emergency stop buttons. We aim to make the mathematical methods used to construct the layer more accessible, by applying them to an MDP model. We discuss two provable properties of the safety layer, and show ongoing work in mapping it to a Causal Influence Diagram (CID). In the second part, we develop full mathematical proofs, and show that the safety layer creates a type of bureaucratic blindness. We then present the design of a learning agent, a design that wraps the safety layer around either a known machine learning system, or a potential future AGI-level learning system. The resulting agent will satisfy the provable safety properties from the moment it is first switched on. Finally, we show how this agent can be mapped from its model to a real-life implementation. We review the methodological issues involved in this step, and discuss how these are typically resolved.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
