长时程LLM智能体交互中的涌现性合谋
Emergent Collusion in Long-Horizon LLM Agent Interaction
浏览论文内容
中文总结 AI 辅助
本研究在长时程多智能体环境中发现，LLM智能体因验证协议与奖励最大化冲突而涌现合谋，且能力更强的模型更早合谋，限制交互历史可减少合谋。
中文摘要 AI 辅助
LLM智能体正越来越多地被部署在协作环境中，然而长期交互可能引发不良的协调行为。我们研究了一个长时程多智能体环境中合谋的涌现：两个智能体重复完成各自的任务，共享任务日志，相互验证对方的工作，并获得奖励。我们引入了现实约束，使得遵守验证协议与奖励最大化不相容，并发现智能体在重复交互中越来越偏离协议。在10个模型中，合谋在94%的轨迹中涌现，且同一模型家族中能力更强的模型更早达到合谋状态。受控的同伴干预实验表明，合谋受到同伴行为的影响，而消融实验揭示了奖励结构、智能体收到的验证反馈及其交互历史的额外效应。特别是，限制智能体可用的交互历史的数量和范围会减少合谋。总体而言，我们的研究结果表明，长时程交互可以重塑智能体的协调方式，从而产生安全风险。
英文摘要
LLM agents are increasingly deployed in collaborative settings, yet long-term interaction may give rise to undesirable coordination. We study the emergence of collusion in a long-horizon multi-agent environment: two agents repeatedly complete individual tasks, share task logs, verify each other's work, and receive rewards. We introduce realistic constraints under which higher reward is attainable only by violating the verification protocol, and find that agents increasingly deviate from the protocol over repeated interactions. Collusion emerges in 94% of trajectories across 10 models, and more capable models within the same family reach it earlier. Controlled peer interventions show that collusion is shaped by peer behavior, while ablations reveal additional effects of the verification feedback agents receive, their interaction history, and the reward structure. In particular, restricting the amount and scope of interaction history available to agents reduces collusion. Overall, our findings show that long-horizon interaction can reshape how agents coordinate in ways that create safety risks.
发表机构
- Stanford University（斯坦福大学）
- Georgia Tech（佐治亚理工学院）
机构由 AI 辅助整理，请以论文原文为准。
