正则化强调时序差分学习：常数步长下的稳定性

中文摘要 AI 辅助

强调时序差分学习（ETD）稳定了期望的离策略TD更新并改变了其投影几何，但这两个性质都不能决定常数步长下的采样动力学。我们构造了一个遍历的两状态反例，其中ETD均值映射收缩，而采样乘积具有正的最大李雅普诺夫指数。再生循环分析将此符号与后续迹的无限方差区分开来。我们引入了正则化强调TD（RETD），这是一种归一化的一阶冲击后修复方法，它保持迹和重要性比率不变，将强调TD信号存储在泄漏标量状态中，并释放延迟校正。RETD的原始平衡点是ETD平衡点的仿射平移；单正则化和双正则化读出可精确恢复ETD不动点。我们证明了调和递减步长下的几乎必然收敛性，以及基于马尔可夫随机乘积界的条件常数步长矩收缩结果。RETD在两状态构造和一个Baird点上具有经过认证的负指数，而Baird ETD的正符号仍然是数值性的。配对的10,000次运行实验验证了两种分离、不动点恢复、非单调稳定区域以及任务依赖性。RETD改变了冲击后的动力学；它并未减少共享的后续迹方差。

英文摘要

Emphatic temporal-difference learning (ETD) stabilizes the expected off-policy TD update and changes its projection geometry, but neither property determines constant-stepsize sampled dynamics. We construct an ergodic two-state counterexample in which the ETD mean map contracts while the sampled product has a positive top Lyapunov exponent. Regenerative-cycle analysis separates this sign from the infinite variance of the follow-on trace. We introduce regularized emphatic TD (RETD), a normalized first-order post-shock repair that leaves the trace and importance ratios unchanged, stores the emphatic TD signal in a leaky scalar state, and releases a delayed correction. RETD's raw equilibrium is an affine shift of the ETD equilibrium; single- and two-regularization readouts recover the ETD fixed point exactly. We prove almost-sure convergence for harmonic diminishing stepsizes and a conditional constant-stepsize moment-contraction result from a Markovian random-product bound. RETD has certified negative exponents on the two-state construction and one Baird point, whereas the positive Baird ETD sign remains numerical. Paired 10,000-run experiments validate both separations, fixed-point recovery, a nonmonotone stability region, and task dependence. RETD changes post-shock dynamics; it does not reduce the shared follow-on-trace variance.

发表机构

• Nanjing University of Posts & Telecommunications（南京邮电大学）
• Nanjing University（南京大学）
• Microsoft Corporation（微软公司）
• National University of Defense Technology（国防科技大学）

机构由 AI 辅助整理，请以论文原文为准。