arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.01166cs.MAcs.AIcs.LG
收藏

角色扮演:在多智能体交互中学习自适应的角色特定策略

Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions

Academy for Engineering and Technology, Fudan University(复旦大学工程技术学院);School of New Media and Communication, Tianjin University(天津大学新媒体与传播学院);Ji Hua Laboratory(季华实验室)

Weifan Long, Wen Wen, Peng Zhai, Lihua Zhang

AI总结:

针对多智能体强化学习的零样本协调问题,受社会价值取向启发提出Role Play框架,通过角色嵌入与角色预测器实现角色自适应策略学习,理论证明可达近似最优,在多种游戏中优于强基线。

摘要:

多智能体强化学习(MARL)中的零样本协调问题要求智能体适应未见过的智能体,已受到越来越多的关注。传统方法通常依赖自博弈(Self-Play, SP)框架在策略池中生成多样化的策略集,以此提升最终智能体的泛化能力。然而,这些框架可能难以覆盖全部潜在策略,尤其是在需要智能体平衡合作与竞争的现实场景中。在这类场景下,智能体需要能够适应多变且往往相互冲突的目标的策略。受社会价值取向(Social Value Orientation, SVO)——即个体在与他人互动时保持稳定的价值取向——的启发,我们提出了一种名为\textit{Role Play}(角色扮演,RP)的新型框架。RP利用角色嵌入将策略多样性的挑战转化为更易处理的角色多样性问题。它训练一个带有角色嵌入观测的公共策略,并采用角色预测器估计其他智能体的联合角色嵌入,帮助学习智能体适应其被分配的角色。我们从理论上证明,通过优化相对于基于角色的近似策略的期望累积奖励,可以得到近似最优策略。在合作游戏(Overcooked)和混合动机游戏(Harvest、CleanUp)中的实验结果表明,在与未见过的智能体交互时,RP始终优于强基线方法,凸显了其在复杂环境中的鲁棒性和适应性。

英文摘要:

Zero-shot coordination problem in multi-agent reinforcement learning (MARL), which requires agents to adapt to unseen agents, has attracted increasing attention. Traditional approaches often rely on the Self-Play (SP) framework to generate a diverse set of policies in a policy pool, which serves to improve the generalization capability of the final agent. However, these frameworks may struggle to capture the full spectrum of potential strategies, especially in real-world scenarios that demand agents balance cooperation with competition. In such settings, agents need strategies that can adapt to varying and often conflicting goals. Drawing inspiration from Social Value Orientation (SVO)-where individuals maintain stable value orientations during interactions with others-we propose a novel framework called \emph{Role Play} (RP). RP employs role embeddings to transform the challenge of policy diversity into a more manageable diversity of roles. It trains a common policy with role embedding observations and employs a role predictor to estimate the joint role embeddings of other agents, helping the learning agent adapt to its assigned role. We theoretically prove that an approximate optimal policy can be achieved by optimizing the expected cumulative reward relative to an approximate role-based policy. Experimental results in both cooperative (Overcooked) and mixed-motive games (Harvest, CleanUp) reveal that RP consistently outperforms strong baselines when interacting with unseen agents, highlighting its robustness and adaptability in complex environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑