智能体AI系统中的风险对齐
Risk Alignment in Agentic AI Systems
AI总结：
本文探讨智能体AI的风险对齐问题，分析风险态度对用户信任与社会责任的影响，并提出三篇论文以解决风险态度校准、护栏设置及代他人决策的伦理等规范性与技术性挑战。
摘要：
智能体AI——即有能力且被允许在极少监督下执行复杂行动的AI——标志着AI能力的新前沿，并引发了关于如何安全地创建此类系统并使其与用户、开发者和社会对齐的新问题。由于智能体的行动受其风险态度的影响，对齐的一个关键方面涉及智能体AI的风险画像。风险对齐不仅关系到用户满意度和信任，还将对社会产生更广泛的重要影响，尤其是随着智能体AI变得更加自主并被允许控制我们生活的关键方面。对风险持鲁莽态度的AI（无论是因为它们被校准为匹配鲁莽的人类用户，还是因为设计不良）可能构成重大威胁。它们还可能打开“责任缺口”，即没有任何智能体能够为有害行为承担责任。什么样的风险态度应指导智能体AI的决策？我们如何设计出与用户风险态度相校准的AI系统？对于可允许的风险态度范围，应当设置哪些护栏（如果有的话）？在设计代表他人做出风险决策的系统时，涉及哪些伦理考量？我们呈现三篇论文，它们涉及这些问题的关键规范性和技术性方面。
英文摘要：
Agentic AIs $-$ AIs that are capable and permitted to undertake complex actions with little supervision $-$ mark a new frontier in AI capabilities and raise new questions about how to safely create and align such systems with users, developers, and society. Because agents' actions are influenced by their attitudes toward risk, one key aspect of alignment concerns the risk profiles of agentic AIs. Risk alignment will matter for user satisfaction and trust, but it will also have important ramifications for society more broadly, especially as agentic AIs become more autonomous and are allowed to control key aspects of our lives. AIs with reckless attitudes toward risk (either because they are calibrated to reckless human users or are poorly designed) may pose significant threats. They might also open 'responsibility gaps' in which there is no agent who can be held accountable for harmful actions. What risk attitudes should guide an agentic AI's decision-making? How might we design AI systems that are calibrated to the risk attitudes of their users? What guardrails, if any, should be placed on the range of permissible risk attitudes? What are the ethical considerations involved when designing systems that make risky decisions on behalf of others? We present three papers that bear on key normative and technical aspects of these questions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
