The key challenge in multiagent learning is learning a best response to the behaviour of other agents, which may be non-stationary: if the other agents adapt their strategy as well, the learning target moves. Disparate streams of research have approached non-stationarity from several angles, which make a variety of implicit assumptions that make it hard to keep an overview of the state of the art and to validate the innovation and significance of new works. This survey presents a coherent overview of work that addresses opponent-induced non-stationarity with tools from game theory, reinforcement learning and multi-armed bandits. Further, we reflect on the principle approaches how algorithms model and cope with this non-stationarity, arriving at a new framework and five categories (in increasing order of sophistication): ignore, forget, respond to target models, learn models, and theory of mind. A wide range of state-of-the-art algorithms is classified into a taxonomy, using these categories and key characteristics of the environment (e.g., observability) and adaptation behaviour of the opponents (e.g., smooth, abrupt). To clarify even further we present illustrative variations of one domain, contrasting the strengths and limitations of each category. Finally, we discuss in which environments the different approaches yield most merit, and point to promising avenues of future research.
Dealing with Non-Stationarity in MARL via Trust-Region Decomposition
East China Normal University（华东师范大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency
The University of Sydney（悉尼大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；SenseTime Group LTD（商汤科技集团）；Institute for AI, Peking University（北京大学人工智能研究院）
多智能体强化学习（MARL）存在非平稳性问题，即当多个智能体同时更新策略时，每一次迭代的目标都在不断变化。本文从第一性原理出发，提出了双向动作依赖Q学习（ACE）方法来解决非平稳性问题。ACE发展的核心是顺序决策过程，其中一次只允许一个代理采取行动。在这个过程中，每个代理最大化其价值函数给定的行动所采取的前代理在推理阶段。在学习阶段，每个代理最小化TD错误，这取决于后续代理如何对他们选择的动作做出反应。考虑到双向依赖的设计，ACE有效地将多Agent MDP转化为单Agent MDP。我们通过识别适当的网络表示来制定动作依赖关系，从而在一个向前传递中隐式地计算顺序决策过程，从而实现ACE框架。为了验证ACE，我们将其与两个MARL基准的强基线进行比较。在Google Research Football和StarCraft Multi-Agent Challenge上的实验结果表明，ACE算法的性能明显优于现有算法。特别是在SMAC任务上，ACE在几乎所有的硬地图和超硬地图上都达到了100%的成功率。我们进一步研究广泛的研究问题，ACE，包括扩展，推广，和实用性。提供代码以方便进一步研究。
Factored Adaptation for Non-Stationary Reinforcement Learning
City University of Hong Kong（香港城市大学）；Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Amsterdam（阿姆斯特丹大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
