Cooperation between self-interested individuals is a widespread phenomenon in the natural world, but remains elusive in interactions between artificially intelligent agents. Instead, naive reinforcement learning algorithms typically converge to Pareto-dominated outcomes in even the simplest of social dilemmas. An emerging literature on opponent shaping has demonstrated the ability to reach prosocial outcomes by influencing the learning of other agents. However, such methods differentiate through the learning step of other agents or optimize for meta-game dynamics, which rely on privileged access to opponents' learning algorithms or exponential sample complexity, respectively. To provide a learning rule-agnostic and sample-efficient alternative, we introduce Reciprocators, reinforcement learning agents which are intrinsically motivated to reciprocate the influence of opponents' actions on their returns. This approach seeks to modify other agents' $Q$-values by increasing their return following beneficial actions (with respect to the Reciprocator) and decreasing it after detrimental actions, guiding them towards mutually beneficial actions without directly differentiating through a model of their policy. We show that Reciprocators can be used to promote cooperation in temporally extended social dilemmas during simultaneous learning. Our code is available at https://github.com/johnlyzhou/reciprocator/.
Constructive Conflict-Driven Multi-Agent Reinforcement Learning for Strategic Diversity
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);CRISE, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(复杂系统认知与决策智能重点实验室,中国科学院自动化研究所)
Achieving Collective Welfare in Multi-Agent Reinforcement Learning via Suggestion Sharing
Warwick Manufacturing Group, University of Warwick, Coventry, UK(沃里克制造集团,沃里克大学,科文特里,英国);School of Electrical Engineering and Computer Science, Louisiana State University, USA(电气工程与计算机科学学院,路易斯安那州立大学,美国);Department of Statistics, University of Warwick, Coventry, UK(统计学系,沃里克大学,科文特里,英国);Alan Turing Institute, London, UK(艾伦·图灵研究所,伦敦,英国)
CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agents
University of Science and Technology of China(中国科学技术大学);Microsoft Research(微软研究院);William & Mary(威廉玛丽学院);Georgia Institute of Technology(佐治亚理工学院);Carnegie Mellon University(卡内基梅隆大学)