arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.12872cs.GT
收藏

作为贝叶斯博弈的拜占庭鲁棒协作多智能体强化学习

Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game

Simin Li, Jun Guo, Jingqiao Xiu, Ruixiao Xu, Xin Yu, Jiakai Wang, Aishan Liu, Yaodong Yang, Xianglong Liu

AI总结:

提出 BARDec-POMDP 贝叶斯博弈框架,将拜占庭智能体建模为自然类型并求解鲁棒贝叶斯马尔可夫完美均衡,配合双时间尺度 actor-critic 算法提升 c-MARL 在多种攻击下的协作鲁棒性。

摘要:

在本研究中,我们探讨协作多智能体强化学习(c-MARL)应对拜占庭故障的鲁棒性;在这种故障中,任何智能体都可能因故障或对抗攻击而采取任意的、最坏情况下的行动。为解决任意智能体都可能具有对抗性这一不确定性,我们提出贝叶斯对抗鲁棒 Dec-POMDP(BARDec-POMDP)框架,该框架将拜占庭对手视为由自然决定的类型,并用独立的转移来表示。这使智能体能够基于其对其他智能体类型的后验信念学习策略,从而促进与已识别盟友的协作,并最大限度降低受对抗操纵的脆弱性。我们将 BARDec-POMDP 的最优解定义为事后鲁棒贝叶斯马尔可夫完美均衡,并证明该均衡存在,且弱占优于以往鲁棒 MARL 方法的均衡。为实现该均衡,我们提出一种双时间尺度 actor-critic 算法,该算法在特定条件下几乎必然收敛。在矩阵博弈、基于等级的觅食任务和 StarCraft II 上的实验表明,即使在最坏情况扰动下,我们的方法也能成功习得复杂的微操技能,并自适应地与盟友协同,展现出对非盲目对手、随机盟友、基于观测的攻击以及基于迁移的攻击的韧性。

英文摘要:

In this study, we explore the robustness of cooperative multi-agent reinforcement learning (c-MARL) against Byzantine failures, where any agent can enact arbitrary, worst-case actions due to malfunction or adversarial attack. To address the uncertainty that any agent can be adversarial, we propose a Bayesian Adversarial Robust Dec-POMDP (BARDec-POMDP) framework, which views Byzantine adversaries as nature-dictated types, represented by a separate transition. This allows agents to learn policies grounded on their posterior beliefs about the type of other agents, fostering collaboration with identified allies and minimizing vulnerability to adversarial manipulation. We define the optimal solution to the BARDec-POMDP as an ex post robust Bayesian Markov perfect equilibrium, which we proof to exist and weakly dominates the equilibrium of previous robust MARL approaches. To realize this equilibrium, we put forward a two-timescale actor-critic algorithm with almost sure convergence under specific conditions. Experimentation on matrix games, level-based foraging and StarCraft II indicate that, even under worst-case perturbations, our method successfully acquires intricate micromanagement skills and adaptively aligns with allies, demonstrating resilience against non-oblivious adversaries, random allies, observation-based attacks, and transfer-based attacks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑