现实中的多智能体学习中,反馈延迟是不可避免的。已知它们会严重降低性能,且在延迟反馈下的收敛速率仍然不明确,即使在双线性游戏中也是如此。本文推导了在无约束双线性游戏中,通过额外乐观预测未来奖励的加权乐观梯度上升(WOGDA)算法的线性收敛速率。为了分析该算法,我们将其解释为额外 proximal 点(EPP)的近似,其更新基于比经典 proximal 点(PP)更远的未来奖励。我们的定理表明,标准乐观(预测下一步奖励)在延迟 m 下,经过 t 次迭代后,收敛到均衡的速率为 exp(-Θ(t/m⁵))。此外,采用额外乐观(预测更远的未来奖励)可以容忍更大的步长,并显著加快速率至 exp(-Θ(t/(m² log m)))。我们的实验也显示了额外乐观带来的加速收敛,并与我们的定理定性一致。总之,本文验证了额外乐观是应对反馈延迟导致性能下降的一种有前途的对策。
英文摘要:
Feedback delays are inevitable in real-world multi-agent learning. They are known to severely degrade performance, and the convergence rate under delayed feedback is still unclear, even for bilinear games. This paper derives the rate of linear convergence of Weighted Optimistic Gradient Descent-Ascent (WOGDA), which predicts future rewards with extra optimism, in unconstrained bilinear games. To analyze the algorithm, we interpret it as an approximation of the Extra Proximal Point (EPP), which is updated based on farther future rewards than the classical Proximal Point (PP). Our theorems show that standard optimism (predicting the next-step reward) achieves linear convergence to the equilibrium at a rate $\exp(-Θ(t/m^{5}))$ after $t$ iterations for delay $m$. Moreover, employing extra optimism (predicting farther future reward) tolerates a larger step size and significantly accelerates the rate to $\exp(-Θ(t/(m^{2}\log m)))$. Our experiments also show accelerated convergence driven by the extra optimism and are qualitatively consistent with our theorems. In summary, this paper validates that extra optimism is a promising countermeasure against performance degradation caused by feedback delays.
On the Convergence of Stochastic Extragradient for Bilinear Games using Restarted Iteration Averaging
University of California, Berkeley(加州大学伯克利分校);Johns Hopkins University(约翰斯·霍普金斯大学);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);Microsoft Research(微软研究院)
Johannes Kepler University Linz(林茨约翰内斯·开普勒大学);Institute for Machine Learning(机器学习研究所);LIT AI Lab(LIT人工智能实验室);Institute of Advanced Research in Artificial Intelligence (IARAI)(人工智能高级研究所(IARAI))
A Reduction-based Framework for Sequential Decision Making with Delayed Feedback
Center for Data Science, Peking University(北京大学数据科学中心);Zhejiang Lab(之江实验室);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);University of Washington(华盛顿大学);University of California, Berkeley(加州大学伯克利分校)
Game-Theoretic Machine Unlearning: Mitigating Extra Privacy Leakage
School of Information Engineering, Zhongnan University of Economics and Law(中南财经政法大学信息工程学院);Faculty of Data Science, City University of Macau(澳门城市大学数据科学学院)