arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.09054cs.MA
收藏

具有异质偏好的多智能体强化学习中的无模型约定

Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences

Raphael Köster, Kevin R. McKee, Richard Everett, Laura Weidinger, William S. Isaac, Edward Hughes, Edgar A. Duéñez-Guzmán, Thore Graepel, Matthew Botvinick, Joel Z. Leibo

摘要:

关于惯例的博弈论观点通常基于常识的概念和个人行为的超理性模型。然而,几十年来行为经济学的研究已经对这两个基础的有效性提出了质疑。与此同时,计算神经科学为决策提供了一个现代化的“双过程”解释,其中无模型(MF)强化学习与基于模型(MB)强化学习进行了权衡。前者捕捉习惯性和程序性学习,而后者捕捉通过明确的计划和推理做出的选择。有些公约(如国际条约)可能得到与博弈论和甲基溴账户相一致的认知的支持。然而,惯例的形成也可能通过MF机制发生,如习惯学习;尽管这种可能性尚未得到充分研究。在这里,我们证明了复杂的,大规模的公约可以出现MF学习机制。这表明,一些惯例可能是由习惯性认知而不是外显推理支持的。我们将MF多智能体强化学习应用到一个时空扩展的不完全信息博弈中。在这个博弈中,状态空间的大部分只有通过集体行动才能到达。然而,口味的异质性使得这种协调行动变得困难:多重均衡对所有参与者都是可取的,但子群更喜欢一个特定的均衡。这就产生了一个协调问题,可以通过制定一项公约来解决。我们调查启动和搭便车的子问题,以及群体规模的影响,强度的内在偏好,和显着的协调公约的出现动态。我们的模拟结果表明,代理建立和转换之间的公约,甚至反对自己的首选结果时,这样做是必要的有效协调。

英文摘要:

Game theoretic views of convention generally rest on notions of common knowledge and hyper-rational models of individual behavior. However, decades of work in behavioral economics have questioned the validity of both foundations. Meanwhile, computational neuroscience has contributed a modernized 'dual process' account of decision-making where model-free (MF) reinforcement learning trades off with model-based (MB) reinforcement learning. The former captures habitual and procedural learning while the latter captures choices taken via explicit planning and deduction. Some conventions (e.g. international treaties) are likely supported by cognition that resonates with the game theoretic and MB accounts. However, convention formation may also occur via MF mechanisms like habit learning; though this possibility has been understudied. Here, we demonstrate that complex, large-scale conventions can emerge from MF learning mechanisms. This suggests that some conventions may be supported by habit-like cognition rather than explicit reasoning. We apply MF multi-agent reinforcement learning to a temporo-spatially extended game with incomplete information. In this game, large parts of the state space are reachable only by collective action. However, heterogeneity of tastes makes such coordinated action difficult: multiple equilibria are desirable for all players, but subgroups prefer a particular equilibrium over all others. This creates a coordination problem that can be solved by establishing a convention. We investigate start-up and free rider subproblems as well as the effects of group size, intensity of intrinsic preference, and salience on the emergence dynamics of coordination conventions. Results of our simulations show agents establish and switch between conventions, even working against their own preferred outcome when doing so is necessary for effective coordination.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑