AGI系统中的反事实规划
Counterfactual Planning in AGI Systems
摘要：
我们提出反事实规划作为一种设计方法，用于创建一系列安全机制，这些机制可以应用于具有人工通用智能的假设未来AI系统。反事实规划的关键步骤是使用AGI机器学习系统来构建一个反事实的世界模型，该模型被设计成与系统所处的真实世界不同。一个反事实的规划代理确定的行动，最大化预期效用在这个反事实的规划世界，然后在现实世界中执行相同的行动。我们使用反事实规划来构建AGI代理紧急停止按钮，以及在智能爆炸之前自动停止代理的安全联锁。我们还构建了一个代理的输入终端，可用于人类迭代地提高代理的奖励功能，其中的激励代理操纵这个改进过程被抑制。作为非智能体AGI系统中反事实规划的一个例子，我们构建了一个反事实预言机。作为一种设计方法，反事实计划是围绕使用图形符号来定义数学反事实而建立的。这种两图表示法还提供了一种紧凑且可读的语言，用于推理通常存在于机器学习代理内部的复杂类型的自引用和间接表示。
英文摘要：
We present counterfactual planning as a design approach for creating a range of safety mechanisms that can be applied in hypothetical future AI systems which have Artificial General Intelligence. The key step in counterfactual planning is to use an AGI machine learning system to construct a counterfactual world model, designed to be different from the real world the system is in. A counterfactual planning agent determines the action that best maximizes expected utility in this counterfactual planning world, and then performs the same action in the real world. We use counterfactual planning to construct an AGI agent emergency stop button, and a safety interlock that will automatically stop the agent before it undergoes an intelligence explosion. We also construct an agent with an input terminal that can be used by humans to iteratively improve the agent's reward function, where the incentive for the agent to manipulate this improvement process is suppressed. As an example of counterfactual planning in a non-agent AGI system, we construct a counterfactual oracle. As a design approach, counterfactual planning is built around the use of a graphical notation for defining mathematical counterfactuals. This two-diagram notation also provides a compact and readable language for reasoning about the complex types of self-referencing and indirect representation which are typically present inside machine learning agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
