arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09509cs.MA
收藏

学习助推以实现条件合作:一种多智能体强化学习模型

Learning Nudges for Conditional Cooperation: A Multi-Agent Reinforcement Learning Model

Shatayu Kulkarni, Sabine Brunswicker

AI总结:

本文提出多智能体强化学习模型,让深度强化学习智能体通过助推激励3名条件合作者合作;实验表明两种奖励设计均显著提升贡献总额与合作比例。

摘要:

公共物品博弈描述了一种社会困境,其中很大一部分智能体表现为条件合作者(CC):只有在看到他人合作时,它们才会采取合作行为,因为它们满足于与他人行为保持一致的社会规范,而不是对合作进行优化。条件合作者由基于抱负的强化学习所引导,该学习以过去与他人互动的经验和令人满意的抱负为指导。在许多现实环境中,强化社会规范并不会自发形成。本文提出,一个进行优化的强化智能体可以通过助推(即促使合作发生的间接机制)来促进合作。该智能体的目标是通过自身行动激励条件合作者进行合作,从而创造出表明他人正在合作的社会规范。我们引入了一种用于公共物品博弈的多智能体强化学习模型,其中包含3个使用抱负型强化学习的CC学习智能体,以及1个使用深度强化学习来学习可优化合作的助推行为的助推智能体。对于助推智能体,我们建模了两种不同的奖励函数:一种最大化博弈总回报(sum DRL),另一种最大化高于比例阈值的合作贡献数量(prop DRL)。结果表明,我们用于CC智能体的基于抱负的RL模型与经验观察到的CC行为一致。由3个CC RL智能体和1个助推RL智能体组成的博弈,优于仅由4个CC RL智能体组成的基线。sum DRL助推智能体使贡献总额提高8.22%,使合作贡献的总比例提高12.42%;而prop DRL助推智能体使贡献总额提高8.85%,使合作贡献的总比例提高14.87%。我们的发现推进了公共物品博弈和强化学习领域的文献。

英文摘要:

The public goods game describes a social dilemma in which a large proportion of agents act as conditional cooperators (CC): they only act cooperatively if they see others acting cooperatively because they satisfice with the social norm to be in line with what others are doing instead of optimizing cooperation. CCs are guided by aspiration-based reinforcement learning guided by past experiences of interactions with others and satisficing aspirations. In many real-world settings, reinforcing social norms do not emerge. In this paper, we propose that an optimizing reinforcement agent can facilitate cooperation through nudges, i.e. indirect mechanisms for cooperation to happen. The agent's goal is to motivate CCs into cooperation through its own actions to create social norms that signal that others are cooperating. We introduce a multi-agent reinforcement learning model for public goods games, with 3 CC learning agents using aspirational reinforcement learning and 1 nudging agent using deep reinforcement learning to learn nudges that optimize cooperation. For our nudging agent, we model two distinct reward functions, one maximizing the total game return (sum DRL) and one maximizing the number of cooperative contributions contributions higher than a proportional threshold (prop DRL). Our results show that our aspiration-based RL model for CC agents is consistent with empirically observed CC behavior. Games combining 3 CC RL agents and one nudging RL agent outperform the baseline consisting of 4 CC RL agents only. The sum DRL nudging agent increases the total sum of contributions by 8.22% and the total proportion of cooperative contributions by 12.42%, while the prop nudging DRL increases the total sum of contributions by 8.85% and the total proportion of cooperative contributions by 14.87%. Our findings advance the literature on public goods games and reinforcement learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑