意图对齐的人工智能系统会削弱人类自主性：人工智能安全需要自主性基础研究
Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety
AI总结：
本文指出仅靠意图对齐不足以确保AI安全，提出以保护人类长期自主性为更稳健标准，首次形式化定义保持自主性的AI-人类交互，并倡议建立“agency foundations”研究领域。
摘要：
人工智能（AI）系统的快速发展表明，通用人工智能（AGI）系统可能很快就会出现。许多研究者担心，AI和AGI会通过故意滥用（AI-misuse）或意外事故（AI-accidents）对人类造成伤害。针对AI-accidents，越来越多的研究致力于开发算法和范式，以确保AI系统与人类的意图保持一致，例如，使AI系统产生的行动或建议能够被人类判断为与其意图和目标一致。本文认为，与人类意图对齐对于安全的AI系统而言是不够的，而保护人类的长期自主性（agency）可能是一个更稳健的标准，并且这一标准需要在优化过程中被明确地、先验地分离出来。我们认为，AI系统可以重塑人类意图，并讨论了缺乏保护人类免受自主性丧失的生物学和心理学机制。我们首次给出了保持自主性的AI-人类交互的形式化定义，该定义侧重于前瞻性的自主性评估，并主张必须越来越多地由AI系统——而非人类——来承担这些评估任务。我们展示了在包含使用时间差分学习（temporal-difference learning）来提出行动建议的嵌入式智能体的简单环境中，自主性丧失是如何发生的。最后，我们提出了一个名为“agency foundations”的新研究领域，并提出了四个初始主题，旨在增进我们对AI-人类交互中自主性的理解：仁慈博弈论（benevolent game theory）、人权的算法基础（algorithmic foundations of human rights）、神经网络中自主性表征的机制可解释性（mechanistic interpretability of agency representation in neural-networks），以及基于内部状态的强化学习（reinforcement learning from internal states）。
英文摘要：
The rapid advancement of artificial intelligence (AI) systems suggests that artificial general intelligence (AGI) systems may soon arrive. Many researchers are concerned that AIs and AGIs will harm humans via intentional misuse (AI-misuse) or through accidents (AI-accidents). In respect of AI-accidents, there is an increasing effort focused on developing algorithms and paradigms that ensure AI systems are aligned to what humans intend, e.g. AI systems that yield actions or recommendations that humans might judge as consistent with their intentions and goals. Here we argue that alignment to human intent is insufficient for safe AI systems and that preservation of long-term agency of humans may be a more robust standard, and one that needs to be separated explicitly and a priori during optimization. We argue that AI systems can reshape human intention and discuss the lack of biological and psychological mechanisms that protect humans from loss of agency. We provide the first formal definition of agency-preserving AI-human interactions which focuses on forward-looking agency evaluations and argue that AI systems - not humans - must be increasingly tasked with making these evaluations. We show how agency loss can occur in simple environments containing embedded agents that use temporal-difference learning to make action recommendations. Finally, we propose a new area of research called "agency foundations" and pose four initial topics designed to improve our understanding of agency in AI-human interactions: benevolent game theory, algorithmic foundations of human rights, mechanistic interpretability of agency representation in neural-networks and reinforcement learning from internal states.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
