arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 0911.2406physics.soc-phnlin.AOphysics.comp-ph
收藏

多智能体强化学习的无标度记忆模型。平均场近似与石头剪刀布动力学

Scale-free memory model for multiagent reinforcement learning. Mean field approximation and rock-paper-scissors dynamics

Ihor Lubashevsky, Shigeru Kanemoto

摘要:

提出了一种基于无标度记忆强化学习的多智能体系统的连续时间模型。代理人被假定为相互独立的行动,通过试错搜索优化他们的选择可能的行动。为了获得对行动价值的认识,智能体在记忆中积累了在每个时刻采取特定行动所获得的奖励。过去的奖励的贡献,代理当前的动作值的感知描述了一个积分算子与幂律内核。最后得到了控制系统动力学的分数阶微分方程。代理被认为是通过一个代理的奖励,根据其他代理的选择隐式地相互作用。采用成对相互作用模型来描述这种效应。作为非传递性交互系统的一个具体例子,对两Agent和三Agent的石头剪刀布系统进行了详细的分析,包括稳定性分析和数值模拟。无标度记忆被证明会导致系统的复杂动态。特别是,它表明,可以同时有两种模式的系统不稳定性进行亚临界和超临界分岔,与后者表现出异常振荡的振幅和周期随时间的增长。此外,通过这种超临界模式的不稳定性的开始可以被视为“利他主义自组织”。对于三剂系统的不稳定动力学被发现是相当不规则的,可以由交替的片段的振荡不同的性质。

英文摘要:

A continuous time model for multiagent systems governed by reinforcement learning with scale-free memory is developed. The agents are assumed to act independently of one another in optimizing their choice of possible actions via trial-and-error search. To gain awareness about the action value the agents accumulate in their memory the rewards obtained from taking a specific action at each moment of time. The contribution of the rewards in the past to the agent current perception of action value is described by an integral operator with a power-law kernel. Finally a fractional differential equation governing the system dynamics is obtained. The agents are considered to interact with one another implicitly via the reward of one agent depending on the choice of the other agents. The pairwise interaction model is adopted to describe this effect. As a specific example of systems with non-transitive interactions, a two agent and three agent systems of the rock-paper-scissors type are analyzed in detail, including the stability analysis and numerical simulation. Scale-free memory is demonstrated to cause complex dynamics of the systems at hand. In particular, it is shown that there can be simultaneously two modes of the system instability undergoing subcritical and supercritical bifurcation, with the latter one exhibiting anomalous oscillations with the amplitude and period growing with time. Besides, the instability onset via this supercritical mode may be regarded as "altruism self-organization". For the three agent system the instability dynamics is found to be rather irregular and can be composed of alternate fragments of oscillations different in their properties.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑