在线学习无信息马尔可夫游戏：经验纳什价值遗憾与非平稳适应
Online Learning for Uninformed Markov Games: Empirical Nash-Value Regret and Non-Stationarity Adaptation
AI总结：
本文提出了一种参数无关算法，通过经验纳什价值遗憾度量，在对手非平稳性适应下实现O(min{√K + (CK)^{1/3},√LK})的遗憾界。
摘要：
我们研究了两名玩家无信息马尔可夫游戏中的在线学习，其中对手的动作和策略是未知的。在这一设定中，Tian等人（2021）表明，在不产生指数依赖于回合长度H的情况下，无法实现无外部遗憾。他们随后转向更弱的纳什价值遗憾概念，并提出了一种V学习算法，其遗憾为O(K^{2/3})在K个回合后。然而，他们的算法和保证并不适应问题的难度：即使在对手遵循固定策略的情况下，O(√K)外部遗憾众所周知可以实现，他们的结果仍然是更弱度量下的更差率O(K^{2/3})。在本工作中，我们全面解决了这两个限制。首先，我们引入了经验纳什价值遗憾，这是一种严格强于纳什价值遗憾的新遗憾概念，并且在对手遵循固定策略时自然退化为外部遗憾。此外，在这个新度量下，我们提出了一种参数无关算法，其遗憾界为O(min{√K + (CK)^{1/3},√LK})，其中C量化了对手策略的方差，L表示策略切换次数（两者均至多O(K))。因此，我们的结果不仅恢复了两个极端——当对手固定时O(√K)外部遗憾和最坏情况下的O(K^{2/3})纳什价值遗憾，还通过自动适应对手的非平稳性在两者之间平滑插值。我们通过首先提供新的epoch-based V学习算法分析，由Mao等人（2022）建立O(ηC + √K/η)的遗憾界，其中η是epoch增量因子。然后，我们展示如何根据对手潜在的非平稳性适配性重新启动此算法，最终达到我们的最终结果。
英文摘要：
We study online learning in two-player uninformed Markov games, where the opponent's actions and policies are unobserved. In this setting, Tian et al. (2021) show that achieving no-external-regret is impossible without incurring an exponential dependence on the episode length $H$. They then turn to the weaker notion of Nash-value regret and propose a V-learning algorithm with regret $O(K^{2/3})$ after $K$ episodes. However, their algorithm and guarantee do not adapt to the difficulty of the problem: even in the case where the opponent follows a fixed policy and thus $O(\sqrt{K})$ external regret is well-known to be achievable, their result is still the worse rate $O(K^{2/3})$ on a weaker metric. In this work, we fully address both limitations. First, we introduce empirical Nash-value regret, a new regret notion that is strictly stronger than Nash-value regret and naturally reduces to external regret when the opponent follows a fixed policy. Moreover, under this new metric, we propose a parameter-free algorithm that achieves an $O(\min \{\sqrt{K} + (CK)^{1/3},\sqrt{LK}\})$ regret bound, where $C$ quantifies the variance of the opponent's policies and $L$ denotes the number of policy switches (both at most $O(K)$). Therefore, our results not only recover the two extremes -- $O(\sqrt{K})$ external regret when the opponent is fixed and $O(K^{2/3})$ Nash-value regret in the worst case -- but also smoothly interpolate between these extremes by automatically adapting to the opponent's non-stationarity. We achieve so by first providing a new analysis of the epoch-based V-learning algorithm by Mao et al. (2022), establishing an $O(ηC + \sqrt{K/η})$ regret bound, where $η$ is the epoch incremental factor. Next, we show how to adaptively restart this algorithm with an appropriate $η$ in response to the potential non-stationarity of the opponent, eventually achieving our final results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
