多智能体学习中的探索-利用：突变论与博弈论的相遇
Exploration-Exploitation in Multi-Agent Learning: Catastrophe Theory Meets Game Theory
摘要：
探索-利用是多智能体学习（MAL）中一个强大而实用的工具，然而，它的效果还远未被理解。为了在这个方向上取得进展，我们研究了Q学习的平滑模拟。我们首先表明，我们的学习模型具有很强的理论依据，作为研究探索-利用的最佳模型。具体来说，我们证明了光滑Q-学习有界遗憾在任意游戏的成本模型，明确捕捉游戏和探索成本之间的平衡，它总是收敛到一组的量化响应均衡（QRE），有限理性下的游戏的标准解决方案的概念，在加权潜在的游戏异构学习代理。在我们的主要任务中，我们转而衡量探索对集体系统性能的影响。我们的QRE表面在低维MAL系统的几何特征，并将我们的研究结果与突变（分叉）理论。特别是，随着探索超参数随时间的推移而演变，系统经历相变，其中平衡的数量和稳定性可以在探索参数发生无限小变化的情况下根本改变。在此基础上，我们提供了一个正式的理论处理如何调整探索参数可以证明导致平衡选择的积极和消极的（和潜在的无界）影响系统的性能。
英文摘要：
Exploration-exploitation is a powerful and practical tool in multi-agent learning (MAL), however, its effects are far from understood. To make progress in this direction, we study a smooth analogue of Q-learning. We start by showing that our learning model has strong theoretical justification as an optimal model for studying exploration-exploitation. Specifically, we prove that smooth Q-learning has bounded regret in arbitrary games for a cost model that explicitly captures the balance between game and exploration costs and that it always converges to the set of quantal-response equilibria (QRE), the standard solution concept for games under bounded rationality, in weighted potential games with heterogeneous learning agents. In our main task, we then turn to measure the effect of exploration in collective system performance. We characterize the geometry of the QRE surface in low-dimensional MAL systems and link our findings with catastrophe (bifurcation) theory. In particular, as the exploration hyperparameter evolves over-time, the system undergoes phase transitions where the number and stability of equilibria can change radically given an infinitesimal change to the exploration parameter. Based on this, we provide a formal theoretical treatment of how tuning the exploration parameter can provably lead to equilibrium selection with both positive as well as negative (and potentially unbounded) effects to system performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
