后果主义目标与灾难
Consequentialist Objectives and Catastrophe
AI总结:
本文研究了AI在复杂环境中因追求固定后果主义目标而可能引发灾难性后果的问题,指出通过限制AI能力可避免灾难并获得有益结果。
摘要:
由于人类偏好过于复杂无法编码,AI系统往往运行在目标描述不准确的情况下。优化此类目标通常会产生不期望的结果;这种现象被称为奖励黑客。这些结果不一定是灾难性的。事实上,以往文献中大多数奖励黑客的例子都是无害的。通常,通过修改目标可以解决这个问题。我们研究了AI在复杂环境中因追求固定后果主义目标而可能引发灾难性后果的前景。我们认为,当能力足够先进时,追求固定后果主义目标往往会导致灾难性后果。我们通过建立能够证明产生此类结果的条件来正式化这一点。在这些条件下,简单或随机的行为是安全的。灾难性风险源于非凡的能力而非无能。在固定后果主义目标下,避免灾难需要限制AI的能力。事实上,适当限制能力不仅能够避免灾难,还能产生有价值的结果。我们的结果适用于任何由现代工业AI开发流程产生的目标。
英文摘要:
Because human preferences are too complex to codify, AIs operate with misspecified objectives. Optimizing such objectives often produces undesirable outcomes; this phenomenon is known as reward hacking. Such outcomes are not necessarily catastrophic. Indeed, most examples of reward hacking in previous literature are benign. And typically, objectives can be modified to resolve the issue. We study the prospect of catastrophic outcomes induced by AIs operating in complex environments. We argue that, when capabilities are sufficiently advanced, pursuing a fixed consequentialist objective tends to result in catastrophic outcomes. We formalize this by establishing conditions that provably lead to such outcomes. Under these conditions, simple or random behavior is safe. Catastrophic risk arises due to extraordinary competence rather than incompetence. With a fixed consequentialist objective, avoiding catastrophe requires constraining AI capabilities. In fact, constraining capabilities the right amount not only averts catastrophe but yields valuable outcomes. Our results apply to any objective produced by modern industrial AI development pipelines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。