零和马氏对策中的可微仲裁
Differentiable Arbitrating in Zero-sum Markov Games
摘要:
本文首先研究了如何在两人零和马尔可夫博弈中扰动报酬,使之达到理想的纳什均衡,即仲裁。这样的问题承认一个双层优化配方。较低的层次要求在给定的奖励函数下求解纳什均衡,这使得整个问题具有端到端优化的挑战性。我们提出了一个反向传播方案,区分通过纳什均衡,它提供了梯度反馈的上层。特别是,我们的方法只需要一个黑盒求解器(正则化)纳什均衡(NE)。我们开发了适当的黑盒NE求解器的收敛性分析,并证明了经验的成功,在两个多智能体强化学习(MARL)环境。
英文摘要:
We initiate the study of how to perturb the reward in a zero-sum Markov game with two players to induce a desirable Nash equilibrium, namely arbitrating. Such a problem admits a bi-level optimization formulation. The lower level requires solving the Nash equilibrium under a given reward function, which makes the overall problem challenging to optimize in an end-to-end way. We propose a backpropagation scheme that differentiates through the Nash equilibrium, which provides the gradient feedback for the upper level. In particular, our method only requires a black-box solver for the (regularized) Nash equilibrium (NE). We develop the convergence analysis for the proposed framework with proper black-box NE solvers and demonstrate the empirical successes in two multi-agent reinforcement learning (MARL) environments.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。