arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03911cs.LGmath.OCstat.ML
收藏

在Q学习中目标更新频率的作用

The Role of Target Update Frequencies in Q-Learning

Institute of Mathematics, University of Mannheim(曼海姆大学数学研究所);Department of Mathematics & Informatics, University of Wuppertal(乌珀塔尔大学数学与信息学系)

Simon Weissmann, Tilman Aach, Benedikt Wille, Sebastian Kassing, Leif Döring

AI总结:

本文通过理论分析揭示了Q学习中目标更新频率对收敛性的影响,证明自适应更新策略能优化样本复杂度。

摘要:

目标网络更新频率(TUF)是(深度)Q学习中的核心稳定机制。然而,其选择仍然缺乏深入理解,通常被当作另一个可调超参数来处理,而非一种原则性设计决策。本文通过近似动态规划的视角,对表格Q学习中的目标固定进行了理论分析。我们将周期性目标更新 formulation 为一种嵌套优化方案,在其中每个外迭代应用一个近似贝尔曼最优性算子,该算子通过通用内循环优化器进行近似。严谨的理论推导为异步采样设置提供了有限时间收敛分析,专门化为内循环中的随机梯度下降。我们的结果提供了由目标更新周期引起的偏差-方差权衡的显式表征,展示了如何最优地设置这个关键超参数。我们证明了恒定的目标更新计划是次优的,在样本复杂度上引入了对数性的开销,这完全可以通过自适应计划来避免。我们的分析表明,最优的目标更新频率在学习过程中以几何方式增加。

英文摘要:

The target network update frequency (TUF) is a central stabilization mechanism in (deep) Q-learning. However, their selection remains poorly understood and is often treated merely as another tunable hyperparameter rather than as a principled design decision. This work provides a theoretical analysis of target fixing in tabular Q-learning through the lens of approximate dynamic programming. We formulate periodic target updates as a nested optimization scheme in which each outer iteration applies an inexact Bellman optimality operator, approximated by a generic inner loop optimizer. Rigorous theory yields a finite-time convergence analysis for the asynchronous sampling setting, specializing to stochastic gradient descent in the inner loop. Our results deliver an explicit characterization of the bias-variance trade-off induced by the target update period, showing how to optimally set this critical hyperparameter. We prove that constant target update schedules are suboptimal, incurring a logarithmic overhead in sample complexity that is entirely avoidable with adaptive schedules. Our analysis shows that the optimal target update frequency increases geometrically over the course of the learning process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑