arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.26752cs.AIcs.LG
收藏

视角游戏:学习协作平衡AI代理的安全性与自主性

The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy

Graduate School of Business(商学院);Stanford University(斯坦福大学)

William Overman, Mohsen Bayati

AI总结:

本文提出了一种视角游戏框架,通过马尔可夫游戏模型平衡AI代理的安全性与自主性,减少开放环境中的安全违规。

摘要:

随着越来越有能力的代理被部署,一个核心的安全挑战是如何在不修改底层系统的情况下保留有意义的人类控制。我们研究了一个最小的控制接口,在其中代理选择是否自主行动(游戏)或延迟(请求),同时人类同时选择是否宽容(信任)或进行监督(监督),并将这种互动建模为一个双人马尔可夫游戏。当这个游戏形成马尔可夫势游戏时,我们证明了一种对齐保证:代理从更自主行动中获得的任何收益增加都不可能减少人类的价值。这建立了一种内在对齐形式,其中代理寻求自主性的激励与人类的福利在结构上耦合。实际上,该框架诱导了一个透明的控制层,鼓励代理在风险时延迟,在安全时行动。尽管我们使用网格世界模拟来说明这种协作的出现,我们的主要验证涉及一个代理工具使用任务,其中两个30B参数语言模型通过独立策略梯度进行微调。我们证明,即使代理在飞行中学习协调,该框架在现实的开放环境中有效减少了安全违规。

英文摘要:

As increasingly capable agents are deployed, a central safety challenge is how to retain meaningful human control without modifying the underlying system. We study a minimal control interface in which an agent chooses whether to act autonomously (play) or defer (ask), while a human simultaneously chooses whether to be permissive (trust) or engage in oversight (oversee), and model this interaction as a two-player Markov game. When this game forms a Markov Potential Game, we prove an alignment guarantee: any increase in the agent's utility from acting more autonomously cannot decrease the human's value. This establishes a form of intrinsic alignment where the agent's incentive to seek autonomy is structurally coupled to the human's welfare. Practically, the framework induces a transparent control layer that encourages the agent to defer when risky and act when safe. While we use gridworld simulations to illustrate the emergence of this collaboration, our primary validation involves an agentic tool-use task in which two 30B parameter language models are fine-tuned via independent policy gradient. We demonstrate that even as the agents learn to coordinate on the fly, this framework effectively reduces safety violations in realistic, open-ended environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑