声誉是促进多智能体系统中合作的有力机制,因为智能体更倾向于与具有良好社会声誉的个体合作。现有的多智能体强化学习方法通常依赖预定义的社会规范来分配声誉,但当智能体持有私有的、独立的观点时,群体如何在评判上达成共识这一问题仍未解决。本文提出了一种新颖的自下而上的声誉学习方法——带声誉奖励的学习(Learning with Reputation Reward,LR2),旨在通过基于所分配声誉的奖励塑形来促进合作行为。我们的智能体架构包含一个困境策略和一个评估策略:困境策略通过考虑对邻居的影响来决定是否合作,评估策略则分配声誉以影响邻居的行为,同时优化自身目标。该方法仅利用局部观测和基于交互的奖励运行,不依赖集中式模块或预定义规范。研究结果表明,LR2 在各种空间社会困境场景中均具有有效性和适应性。有趣的是,我们发现 LR2 不仅通过自下而上声誉的奖励重塑来稳定和增强合作,还通过促进结构化群体中的策略聚类来创造有利于持续合作的环境。
英文摘要:
Reputation serves as a powerful mechanism for promoting cooperation in multi-agent systems, as agents are more inclined to cooperate with those of good social standing. While existing multi-agent reinforcement learning methods typically rely on predefined social norms to assign reputations, the question of how a population reaches a consensus on judgement when agents hold private, independent views remains unresolved. In this paper, we propose a novel bottom-up reputation learning method, Learning with Reputation Reward (LR2), designed to promote cooperative behaviour through rewards shaping based on assigned reputation. Our agent architecture includes a dilemma policy that determines cooperation by considering the impact on neighbours, and an evaluation policy that assigns reputations to affect the actions of neighbours while optimizing self-objectives. It operates using local observations and interaction-based rewards, without relying on centralized modules or predefined norms. Our findings demonstrate the effectiveness and adaptability of LR2 across various spatial social dilemma scenarios. Interestingly, we find that LR2 stabilizes and enhances cooperation not only with reward reshaping from bottom-up reputation but also by fostering strategy clustering in structured populations, thereby creating environments conducive to sustained cooperation.
Reputation as a Solution to Cooperation Collapse in LLM-based MASs
School of Mechanical Engineering, Northwestern Polytechnical University(西北工业大学机械工程学院);School of Cybersecurity, Northwestern Polytechnical University(西北工业大学网络安全学院);School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University(西北工业大学人工智能、光学和电子学(iOPEN)学院);Kyushu University(九州大学);South China University of Technology(华南理工大学);Yunnan University of Finance and Economics(云南财经大学);Northwestern Polytechnical University(西北工业大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Emergent Cooperation under Uncertain Incentive Alignment
University of Groningen(格罗宁根大学);ILLC & IvI, University of Amsterdam(阿姆斯特丹大学逻辑、语言与计算研究所及信息学研究所);Vrije Universiteit Brussels(布鲁塞尔自由大学);Utrecht University(乌得勒支大学)
Evolutionary mechanisms that promote cooperation may not promote social welfare
Teesside University(提赛德大学);University of Birmingham(伯明翰大学);University of Maribor(马里博尔大学);Community Healthcare Center Dr. Adolf Drolc Maribor(阿道夫·德罗尔奇博士马里博尔社区医疗中心);Complexity Science Hub Vienna(维也纳复杂科学中心);Kyung Hee University(庆熙大学)