用目标网络打破致命三角
Breaking the Deadly Triad with a Target Network
AI总结：
针对离策略学习、函数逼近与自举导致的致命三角不稳定问题，提出带双投影的目标网络更新规则，并结合岭正则化证明多类线性 TD 与 Q-learning 算法收敛。
摘要：
致命三角指的是强化学习算法在同时采用离策略学习、函数逼近和自举时出现的不稳定性。本文研究将目标网络作为打破致命三角的工具，为目标网络能够稳定训练这一传统经验提供理论支持。我们首先提出并分析一种新的目标网络更新规则，该规则通过两个投影增强常用的 Polyak 平均式更新。随后，我们将目标网络和岭正则化应用于若干发散算法，并证明它们收敛到正则化 TD 不动点。这些算法是结合线性函数逼近与自举的离策略算法，涵盖策略评估和控制，以及折扣和平均回报设置。特别地，我们首次给出了在非限制性且不断变化的行为策略下、无需双层优化的收敛线性 Q-learning 算法。
英文摘要：
The deadly triad refers to the instability of a reinforcement learning algorithm when it employs off-policy learning, function approximation, and bootstrapping simultaneously. In this paper, we investigate the target network as a tool for breaking the deadly triad, providing theoretical support for the conventional wisdom that a target network stabilizes training. We first propose and analyze a novel target network update rule which augments the commonly used Polyak-averaging style update with two projections. We then apply the target network and ridge regularization in several divergent algorithms and show their convergence to regularized TD fixed points. Those algorithms are off-policy with linear function approximation and bootstrapping, spanning both policy evaluation and control, as well as both discounted and average-reward settings. In particular, we provide the first convergent linear $Q$-learning algorithms under nonrestrictive and changing behavior policies without bi-level optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
