arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.17486cs.LGcs.GTcs.MAmath.OC
收藏

在延迟反馈游戏中通过额外预测实现线性收敛

Linear Convergence in Games with Delayed Feedback via Extra Prediction

Soken University, Kanagawa, Japan(神户大学)

Yuma Fujimoto, Kenshi Abe, Kaito Ariu

AI总结:

本文提出通过额外乐观预测在延迟反馈游戏中实现线性收敛的方法,证明其在双线性游戏中具有更快的收敛速率。

摘要:

现实中的多智能体学习中,反馈延迟是不可避免的。已知它们会严重降低性能,且在延迟反馈下的收敛速率仍然不明确,即使在双线性游戏中也是如此。本文推导了在无约束双线性游戏中,通过额外乐观预测未来奖励的加权乐观梯度上升(WOGDA)算法的线性收敛速率。为了分析该算法,我们将其解释为额外 proximal 点(EPP)的近似,其更新基于比经典 proximal 点(PP)更远的未来奖励。我们的定理表明,标准乐观(预测下一步奖励)在延迟 m 下,经过 t 次迭代后,收敛到均衡的速率为 exp(-Θ(t/m⁵))。此外,采用额外乐观(预测更远的未来奖励)可以容忍更大的步长,并显著加快速率至 exp(-Θ(t/(m² log m)))。我们的实验也显示了额外乐观带来的加速收敛,并与我们的定理定性一致。总之,本文验证了额外乐观是应对反馈延迟导致性能下降的一种有前途的对策。

英文摘要:

Feedback delays are inevitable in real-world multi-agent learning. They are known to severely degrade performance, and the convergence rate under delayed feedback is still unclear, even for bilinear games. This paper derives the rate of linear convergence of Weighted Optimistic Gradient Descent-Ascent (WOGDA), which predicts future rewards with extra optimism, in unconstrained bilinear games. To analyze the algorithm, we interpret it as an approximation of the Extra Proximal Point (EPP), which is updated based on farther future rewards than the classical Proximal Point (PP). Our theorems show that standard optimism (predicting the next-step reward) achieves linear convergence to the equilibrium at a rate $\exp(-Θ(t/m^{5}))$ after $t$ iterations for delay $m$. Moreover, employing extra optimism (predicting farther future reward) tolerates a larger step size and significantly accelerates the rate to $\exp(-Θ(t/(m^{2}\log m)))$. Our experiments also show accelerated convergence driven by the extra optimism and are qualitatively consistent with our theorems. In summary, this paper validates that extra optimism is a promising countermeasure against performance degradation caused by feedback delays.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑