arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.04357math.OC
收藏

为什么双时间尺度Q学习收敛到不同的平均场解?一个统一收敛性分析

Why does the two-timescale Q-learning converge to different mean field solutions? A unified convergence analysis

Jing An, Jianfeng Lu, Yue Wu, Yang Xiang

AI总结:

本文为统一双时间尺度Q学习算法在固定学习率下收敛到不同平均场解的现象提供统一理论解释,通过构造Lyapunov函数实现全学习率范围的统一收敛分析。

摘要:

我们重新审视了由Angiuli等人最初提出的统一双时间尺度Q学习算法(见引用\cite{angiuli2022unified})。该算法通过分别调整平均场分布和Q函数对应的两个学习率之比,即可有效求解平均场博弈(MFG)和平均场控制(MFC)问题。在本文中,我们针对固定学习率下算法出现的分叉数值结果,提供了全面的理论解释。我们通过建立一个将连续时间平均场问题与其离散时间Q函数对应物关联起来的图解来实现这一目标,该图解构成了算法的基础。我们的关键贡献在于构造了一个同时整合平均场分布和Q函数迭代的Lyapunov函数。该Lyapunov函数促进了算法在整个学习率谱上的统一收敛,从而提供了一个连贯的分析框架。

英文摘要:

We revisit the unified two-timescale Q-learning algorithm as initially introduced by Angiuli et al. \cite{angiuli2022unified}. This algorithm demonstrates efficacy in solving mean field game (MFG) and mean field control (MFC) problems, simply by tuning the ratio of two learning rates for mean field distribution and the Q-functions respectively. In this paper, we provide a comprehensive theoretical explanation of the algorithm's bifurcated numerical outcomes under fixed learning rates. We achieve this by establishing a diagram that correlates continuous-time mean field problems to their discrete-time Q-function counterparts, forming the basis of the algorithm. Our key contribution lies in the construction of a Lyapunov function integrating both mean field distribution and Q-function iterates. This Lyapunov function facilitates a unified convergence of the algorithm across the entire spectrum of learning rates, thus providing a cohesive framework for analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑