arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.09535cs.AI
收藏

通过理性策略梯度实现鲁棒且多样化的多智能体学习

Robust and Diverse Multi-Agent Learning via Rational Policy Gradient

UC Berkeley(加州大学伯克利分校);Google Deepmind(谷歌DeepMind)

Niklas Lauffer, Ameesh Shah, Micah Carroll, Sanjit A. Seshia, Stuart Russell, Michael Dennis

AI总结:

针对对抗性优化在协作多智能体场景中易导致智能体自我破坏的问题,提出理性保留策略优化(RPO)形式及理性策略梯度(RPG)方法,确保智能体理性以避免自我破坏,扩展对抗性优化算法并在协作和一般和环境中取得优异性能。

摘要:

在多智能体场景中,明确搜索智能体策略缺陷的对抗性优化算法已成功用于寻找鲁棒且多样化的策略。然而,对抗性优化的成功很大程度上局限于零和场景,因为其在协作场景中的朴素应用会导致关键失效模式:智能体受到非理性激励而自我破坏,阻碍任务完成并停止进一步学习。为解决这一问题,我们提出理性保留策略优化(RPO),这是一种对抗性优化形式,通过确保智能体保持理性(即其策略相对于某些可能的伙伴策略是最优的)来避免自我破坏。为求解RPO,我们开发理性策略梯度(RPG),该方法通过对手塑造技术优化对抗性目标,在原始游戏的修改版本中训练智能体最大化自身奖励。RPG使我们能够扩展多种现有对抗性优化算法,这些算法不再受自我破坏限制,可找到对抗样本、提升鲁棒性和适应性并学习多样化策略。我们通过实验验证,该方法在多个流行协作和一般和环境中取得优异性能。项目页面见https://rational-policy-gradient.github.io。

英文摘要:

Adversarial optimization algorithms that explicitly search for flaws in agents' policies have been successfully applied to finding robust and diverse policies in multi-agent settings. However, the success of adversarial optimization has been largely limited to zero-sum settings because its naive application in cooperative settings leads to a critical failure mode: agents are irrationally incentivized to self-sabotage, blocking the completion of tasks and halting further learning. To address this, we introduce Rationality-preserving Policy Optimization (RPO), a formalism for adversarial optimization that avoids self-sabotage by ensuring agents remain rational--that is, their policies are optimal with respect to some possible partner policy. To solve RPO, we develop Rational Policy Gradient (RPG), which trains agents to maximize their own reward in a modified version of the original game in which we use opponent shaping techniques to optimize the adversarial objective. RPG enables us to extend a variety of existing adversarial optimization algorithms that, no longer subject to the limitations of self-sabotage, can find adversarial examples, improve robustness and adaptability, and learn diverse policies. We empirically validate that our approach achieves strong performance in several popular cooperative and general-sum environments. Our project page can be found at https://rational-policy-gradient.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑