在多智能体学习的研究课题中，混合动机合作是最突出的挑战之一，主要原因是个人目标与集体目标之间的不匹配。前沿研究集中于将领域知识纳入奖励，并引入额外机制来激励合作。然而，这些方法往往面临诸如人工设计工作量大以及缺乏理论基础等不足。为弥补这一差距，我们将混合动机博弈建模为可微博弈，以便更容易阐明朝向合作的学习动态。更具体地说，我们引入了一种名为利他梯度调整（Altruistic Gradient Adjustment，AgA）的新型优化方法，该方法利用梯度调整逐步对齐个体与集体目标。此外，我们从理论上证明，AgA 在考虑个体利益的同时，能够有效地将梯度吸引到集体目标的稳定不动点，并通过经验证据验证了这些论断。我们通过用于测试混合动机协作的基准环境评估了算法 AgA 的有效性，这些环境包括小规模智能体的双人公共品博弈和序贯社会困境博弈 Cleanup 与 Harvest，以及我们在星际争霸 II 游戏中自行开发的大规模环境。
英文摘要：
Among the research topics in multi-agent learning, mixed-motive cooperation is one of the most prominent challenges, primarily due to the mismatch between individual and collective goals. The cutting-edge research is focused on incorporating domain knowledge into rewards and introducing additional mechanisms to incentivize cooperation. However, these approaches often face shortcomings such as the effort on manual design and the absence of theoretical groundings. To close this gap, we model the mixed-motive game as a differentiable game for the ease of illuminating the learning dynamics towards cooperation. More detailed, we introduce a novel optimization method named \textbf{\textit{A}}ltruistic \textbf{\textit{G}}radient \textbf{\textit{A}}djustment (\textbf{\textit{AgA}}) that employs gradient adjustments to progressively align individual and collective objectives. Furthermore, we theoretically prove that AgA effectively attracts gradients to stable fixed points of the collective objective while considering individual interests, and we validate these claims with empirical evidence. We evaluate the effectiveness of our algorithm AgA through benchmark environments for testing mixed-motive collaboration with small-scale agents such as the two-player public good game and the sequential social dilemma games, Cleanup and Harvest, as well as our self-developed large-scale environment in the game StarCraft II.
NOVA-LINCS；Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa（新里斯本大学科学与技术学院）；INESC-ID；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高级技术学院）；ATP-Group（ATP集团）
Emergent collective intelligence from massive-agent cooperation and competition
Tsinghua University, Shenzhen International Graduate School（清华大学深圳国际研究生院）；University of California, San Diego（加州大学圣地亚哥分校）；Georgia Institute of Technology（佐治亚理工学院）
Evolutionary mechanisms that promote cooperation may not promote social welfare
Teesside University（提赛德大学）；University of Birmingham（伯明翰大学）；University of Maribor（马里博尔大学）；Community Healthcare Center Dr. Adolf Drolc Maribor（阿道夫·德罗尔奇博士马里博尔社区医疗中心）；Complexity Science Hub Vienna（维也纳复杂科学中心）；Kyung Hee University（庆熙大学）
