arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.12199cs.LGcs.GTcs.MA
收藏

经验回放创新动力学

Experience-replay Innovative Dynamics

University of Birmingham(伯明翰大学);Khalifa University(哈利法大学)

Tuo Zhang, Leonardo Stella, Julian Barreiro-Gomez

AI总结:

针对多智能体强化学习在非稳定博弈环境中的收敛问题,提出一种基于经验回放且将修订协议作为可调超参数的MARL算法,通过镜像创新动力学轨迹,将MARL理论保证扩展至复制器动力学之外。

摘要:

尽管取得了突破性成功,但多智能体强化学习(MARL)仍面临不稳定性和非平稳性问题。复制器动力学是演化博弈论(EGT)中最著名的模型,为轨迹收敛到纳什均衡提供了理论框架,因此在稳定博弈环境中被用于确保MARL算法的形式化保证。然而,在其他环境中它们表现出相反的行为,这提出了寻找替代方案以确保收敛的问题。相比之下,创新动力学,如Brown-von Neumann-Nash(BNN)或Smith动力学,会产生周期性轨迹,具有近似纳什均衡的潜力。然而,目前尚未有基于这些动力学提出MARL算法的研究。为应对这一挑战,我们开发了一种新颖的基于经验回放的MARL算法,该算法将修订协议作为可调超参数。我们证明,通过适当调整修订协议,算法的行为能够镜像反映这些动力学产生的轨迹。重要的是,我们的贡献提供了一个框架,能够将MARL算法的理论保证扩展到复制器动力学之外。最后,我们用实证结果证实了理论发现。

英文摘要:

Despite its groundbreaking success, multi-agent reinforcement learning (MARL) still suffers from instability and nonstationarity. Replicator dynamics, the most well-known model from evolutionary game theory (EGT), provide a theoretical framework for the convergence of the trajectories to Nash equilibria and, as a result, have been used to ensure formal guarantees for MARL algorithms in stable game settings. However, they exhibit the opposite behavior in other settings, which poses the problem of finding alternatives to ensure convergence. In contrast, innovative dynamics, such as the Brown-von Neumann-Nash (BNN) or Smith, result in periodic trajectories with the potential to approximate Nash equilibria. Yet, no MARL algorithms based on these dynamics have been proposed. In response to this challenge, we develop a novel experience replay-based MARL algorithm that incorporates revision protocols as tunable hyperparameters. We demonstrate, by appropriately adjusting the revision protocols, that the behavior of our algorithm mirrors the trajectories resulting from these dynamics. Importantly, our contribution provides a framework capable of extending the theoretical guarantees of MARL algorithms beyond replicator dynamics. Finally, we corroborate our theoretical findings with empirical results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑