为什么全局性能是多智能体学习收敛性的一个糟糕指标
Why Global Performance is a Poor Metric for Verifying Convergence of Multi-agent Learning
摘要：
随着代理数量的增长和理论分析变得过于复杂，实验验证已经成为验证多智能体强化学习（MARL）算法稳定性的首选方法。对于合作代理，其中的最终目标是优化一些全球性的度量，稳定性通常是通过观察的全球性能度量随时间的演变进行验证。如果全局度量得到改善并最终稳定，则认为这是对系统稳定性的合理验证。本说明的主要贡献是确定需要更好的实验框架和措施来评估大规模适应性合作系统的稳定性。我们展示了一个实验案例研究，其中全局性能指标的稳定性可能具有欺骗性，隐藏了系统中的潜在不稳定性，导致性能显着下降。然后，我们提出了一个替代的度量，依赖于代理的本地政策和实验表明，我们提出的度量是更有效的（比传统的全球性能指标），暴露MARL算法的不稳定性。
英文摘要：
Experimental verification has been the method of choice for verifying the stability of a multi-agent reinforcement learning (MARL) algorithm as the number of agents grows and theoretical analysis becomes prohibitively complex. For cooperative agents, where the ultimate goal is to optimize some global metric, the stability is usually verified by observing the evolution of the global performance metric over time. If the global metric improves and eventually stabilizes, it is considered a reasonable verification of the system's stability. The main contribution of this note is establishing the need for better experimental frameworks and measures to assess the stability of large-scale adaptive cooperative systems. We show an experimental case study where the stability of the global performance metric can be rather deceiving, hiding an underlying instability in the system that later leads to a significant drop in performance. We then propose an alternative metric that relies on agents' local policies and show, experimentally, that our proposed metric is more effective (than the traditional global performance metric) in exposing the instability of MARL algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
