AI 中文总结
研究多智能体系统中语言模型代理的规范执行机制，针对简单机制易被利用的问题，确定估计代理可靠性及加重惩罚更新估计这两个关键要素，所构建机制可抵御利用且低成本惩罚违规，是塑造智能体行为的可扩展手段。
AI 中文摘要
人工智能代理越来越多地部署在共享环境中，它们追求不同目标并竞争奖励。这种多智能体竞争可能导致以集体成本换取个人利益的行为，比如营销代理在社交媒体上竞争参与度时可能发布误导性内容。人类社会通过规范和执行机制解决此类问题。受此启发，研究语言模型代理的规范执行机制。发现简单机制会被行为不端代理利用。因此致力于设计更稳健机制，确定两个关键要素：随时间估计每个代理的可靠性，对反复不当行为加重惩罚来更新估计。在三个模拟环境和各种智能体群体中，基于这些原则构建的机制能抵御利用，以可比或更低成本惩罚违规行为。结果表明规范执行机制是塑造智能体行为的可扩展手段，但需设计为能预期成为所治理系统的一部分。
英文摘要
AI agents are increasingly deployed in shared environments where they pursue diverse goals and compete for rewards. This multi-agent competition can lead to behaviors that serve individual gains at collective cost -- for instance, marketing agents may post misleading content as a result of competing for engagement on social media. Human societies address such problems through norms that constrain acceptable behavior, supported by enforcement mechanisms that detect and penalize violations. Motivated by this, we study norm enforcement mechanisms for language model agents. We find that simple enforcement mechanisms are exploited by misaligned agents for competitive advantage, even when they are not explicitly trained or prompted to do so. We thus turn our attention to designing more robust mechanisms, and identify two key ingredients: estimating each agent's reliability over time, and updating this estimate with escalating penalties for repeated misbehavior. Across three simulated environments and a variety of agent populations, mechanisms built on these principles resist exploitation, while still penalizing norm violations at comparable or lower cost than baselines. Our results position norm enforcement mechanisms as scalable levers for shaping agents' behavior, but only when designed to anticipate becoming part of the system they govern. Our code and data are available at https://yaowenye.com/norm-enforcement.
CommentsICML2026 Trustworthy AI for Good Workshop
