arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.19752cs.MAcs.AIcs.CEcs.CYcs.SYeess.SY
收藏

多智能体系统中的分布安全软标签治理

Soft-Label Governance for Distributional Safety in Multi-Agent Systems

The George Washington University(乔治·华盛顿大学);SWARM AI Safety(SWARM AI安全)

Aizierjiang Aiersilan, Raeli Savitt

AI总结:

本文提出SWARM框架,通过连续概率标签提升多智能体系统安全性和治理效果,展示软指标在检测代理游戏中的应用。

摘要:

多智能体AI系统表现出单个代理无法产生的新兴风险。现有安全框架依赖二元分类代理行为,丢弃了基于代理的评估中的不确定性。我们引入SWARM(系统-wide风险评估多智能体系统),一种模拟框架,将二元好/坏标签替换为软概率标签p=P(v=+1)∈[0,1],实现连续值收益计算、毒性测量和治理干预。SWARM实现了一个模块化治理引擎,具有可配置的杠杆(交易税、断路器、声誉衰减和随机审计),并通过概率指标量化其效果,包括预期毒性E[1−p|接受]和质量差距E[p|接受]−E[p|拒绝]。在七个场景中进行五种子复制,严格治理使福利下降超过40%而无安全提升。同时,激进内部化系统外部性使总福利从基准+262降至-67,而毒性保持不变。断路器需要仔细校准;过于严格的阈值严重减少系统价值,而最佳阈值在适度福利和最小毒性之间取得平衡。配套实验显示软指标能检测通过传统二元评估的自我优化代理的代理游戏。此基础治理层适用于无需修改的活LLM支持代理(Concordia实体、Claude、GPT-4o Mini)。结果表明分布安全需要连续风险指标,治理杠杆校准涉及可量化安全-福利权衡。源代码和项目资源在https://www.swarm-ai.org/公开。

英文摘要:

Multi-agent AI systems exhibit emergent risks that no single agent produces in isolation. Existing safety frameworks rely on binary classifications of agent behavior, discarding the uncertainty inherent in proxy-based evaluation. We introduce SWARM (\textbf{S}ystem-\textbf{W}ide \textbf{A}ssessment of \textbf{R}isk in \textbf{M}ulti-agent systems), a simulation framework that replaces binary good/bad labels with \emph{soft probabilistic labels} $p = P(v{=}+1) \in [0,1]$, enabling continuous-valued payoff computation, toxicity measurement, and governance intervention. SWARM implements a modular governance engine with configurable levers (transaction taxes, circuit breakers, reputation decay, and random audits) and quantifies their effects through probabilistic metrics including expected toxicity $\mathbb{E}[1{-}p \mid \text{accepted}]$ and quality gap $\mathbb{E}[p \mid \text{accepted}] - \mathbb{E}[p \mid \text{rejected}]$. Across seven scenarios with five-seed replication, strict governance reduces welfare by over 40\% without improving safety. In parallel, aggressively internalizing system externalities collapses total welfare from a baseline of $+262$ down to $-67$, while toxicity remains invariant. Circuit breakers require careful calibration; overly restrictive thresholds severely diminish system value, whereas an optimal threshold balances moderate welfare with minimized toxicity. Companion experiments show soft metrics detect proxy gaming by self-optimizing agents passing conventional binary evaluations. This basic governance layer applies to live LLM-backed agents (Concordia entities, Claude, GPT-4o Mini) without modification. Results show distributional safety requires \emph{continuous} risk metrics and governance lever calibration involves quantifiable safety-welfare tradeoffs. Source code and project resources are publicly available at https://www.swarm-ai.org/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑