arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.12988cs.LGcs.MA
收藏

学习多智能体强化学习中智能体的元表征

Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning

Georgia Institute of Technology(佐治亚理工学院);Tencent AI Lab(腾讯人工智能实验室);Tencent Robotics X(腾讯Robotics X实验室);JD Explore Academy(京东探索研究院)

Shenao Zhang, Li Shen, Lei Han, Li Shen

AI总结:

本文提出 MRA,通过多模态潜在策略显式建模博弈共有与特定策略知识,使智能体可跨种群变化的马尔可夫博弈泛化,并从理论与实验上验证其纳什均衡、快速适应及性能提升。

摘要:

在多智能体强化学习中,智能体在单个 Markov Game(MG,马尔可夫博弈)里学到的行为通常局限于给定的智能体数量。通过改变种群规模而产生的每个独立 MG 都可能具有不同的最优联合策略和特定于博弈的知识,而现代多智能体强化学习算法会对这些内容进行独立建模。在本工作中,我们的重点是创建能够在种群变化的 MG 之间泛化的智能体。每个智能体并非学习单模态策略,而是学习一个策略集合,其中包含跨多种博弈的有效策略。为实现这一点,我们提出了 Meta Representations for Agents(MRA,智能体元表征),它显式建模博弈共有和特定于博弈的策略知识。通过用多模态潜在策略表示策略集合,博弈共有的策略知识和多样的策略模式可经由一个迭代优化过程被发现。我们证明,当潜在空间足够大时,通过近似最大化所得的约束互信息目标,这些策略能够在每个评估 MG 中达到 Nash Equilibrium(纳什均衡)。在潜在空间大小有限的实际场景中部署 MRA 时,可以利用一阶梯度信息实现快速适应。大量实验证明了 MRA 在具有挑战性的评估博弈中提升训练性能和泛化能力的有效性。

英文摘要:

In multi-agent reinforcement learning, the behaviors that agents learn in a single Markov Game (MG) are typically confined to the given agent number. Every single MG induced by varying the population may possess distinct optimal joint strategies and game-specific knowledge, which are modeled independently in modern multi-agent reinforcement learning algorithms. In this work, our focus is on creating agents that can generalize across population-varying MGs. Instead of learning a unimodal policy, each agent learns a policy set comprising effective strategies across a variety of games. To achieve this, we propose Meta Representations for Agents (MRA) that explicitly models the game-common and game-specific strategic knowledge. By representing the policy sets with multi-modal latent policies, the game-common strategic knowledge and diverse strategic modes are discovered through an iterative optimization procedure. We prove that by approximately maximizing the resulting constrained mutual information objective, the policies can reach Nash Equilibrium in every evaluation MG when the latent space is sufficiently large. When deploying MRA in practical settings with limited latent space sizes, fast adaptation can be achieved by leveraging the first-order gradient information. Extensive experiments demonstrate the effectiveness of MRA in improving training performance and generalization ability in challenging evaluation games.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑