arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.01641cs.MAcs.AI
收藏

互惠奖励影响促进自利智能体的合作

Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents

University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

John L. Zhou, Weizhe Hong, Jonathan C. Kao

AI总结:

针对自利AI智能体难以合作的问题,提出不依赖对手学习规则、样本高效的Reciprocators强化学习智能体,通过互惠式调整对手收益引导互利行为,可在同时学习的时序扩展社会困境中促进合作。

摘要:

自利个体间的合作是自然界中普遍存在的现象,但在人工智能智能体的交互中仍难以实现。相反,即使在最简单的社会困境中,朴素的强化学习算法通常也会收敛到帕累托占优的结果。新兴的对手塑造研究已证明,通过影响其他智能体的学习能够达成亲社会结果。但这类方法要么通过其他智能体的学习步骤求导,要么针对元博弈动态进行优化,分别依赖于对对手学习算法的特权访问,或是指数级的样本复杂度。为提供一种不依赖学习规则且样本高效的替代方案,我们提出了Reciprocators(互惠者)——一类具有内在动机去回报对手行动对自身收益影响的强化学习智能体。该方法通过在对手采取(对互惠者而言的)有益行动后提高其收益、采取有害行动后降低其收益,以此修改其他智能体的$Q$值,引导其走向互利行动,且无需通过对手的策略模型直接求导。我们证明,在同时学习的时序扩展社会困境中,Reciprocators可用于促进合作。我们的代码可在https://github.com/johnlyzhou/reciprocator/ 获取。

英文摘要:

Cooperation between self-interested individuals is a widespread phenomenon in the natural world, but remains elusive in interactions between artificially intelligent agents. Instead, naive reinforcement learning algorithms typically converge to Pareto-dominated outcomes in even the simplest of social dilemmas. An emerging literature on opponent shaping has demonstrated the ability to reach prosocial outcomes by influencing the learning of other agents. However, such methods differentiate through the learning step of other agents or optimize for meta-game dynamics, which rely on privileged access to opponents' learning algorithms or exponential sample complexity, respectively. To provide a learning rule-agnostic and sample-efficient alternative, we introduce Reciprocators, reinforcement learning agents which are intrinsically motivated to reciprocate the influence of opponents' actions on their returns. This approach seeks to modify other agents' $Q$-values by increasing their return following beneficial actions (with respect to the Reciprocator) and decreasing it after detrimental actions, guiding them towards mutually beneficial actions without directly differentiating through a model of their policy. We show that Reciprocators can be used to promote cooperation in temporally extended social dilemmas during simultaneous learning. Our code is available at https://github.com/johnlyzhou/reciprocator/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑