周期性正则化Q学习
Periodic Regularized Q-Learning
AI总结：
本文提出周期性正则化Q学习算法，通过正则化投影算子实现稳定收敛，证明了在线性函数逼近下的有限时间收敛性。
摘要：
在强化学习（RL）中，Q学习是一个基本算法，其在表格设置下的收敛性得到了保证。然而，这种收敛性保证在使用线性函数逼近时并不成立。为克服这一限制，大量研究引入了正则化技术以确保在函数逼近下的稳定收敛。在本工作中，我们提出了一种新的算法，周期性正则化Q学习（PRQ）。我们首先在投影算子的层面引入正则化，并显式构造了正则化投影值迭代（RP-VI），随后将其扩展为基于样本的RL算法。通过适当正则化投影算子，所得到的投影值迭代成为收缩映射。通过将这种正则化的投影扩展到随机环境，我们建立了PRQ算法，并提供了严格的理论分析，证明了在使用线性函数逼近时PRQ的有限时间收敛性保证。
英文摘要：
In reinforcement learning (RL), Q-learning is a fundamental algorithm whose convergence is guaranteed in the tabular setting. However, this convergence guarantee does not hold under linear function approximation. To overcome this limitation, a significant line of research has introduced regularization techniques to ensure stable convergence under function approximation. In this work, we propose a new algorithm, periodic regularized Q-learning (PRQ). We first introduce regularization at the level of the projection operator and explicitly construct a regularized projected value iteration (RP-VI), subsequently extending it to a sample-based RL algorithm. By appropriately regularizing the projection operator, the resulting projected value iteration becomes a contraction. By extending this regularized projection into the stochastic setting, we establish the PRQ algorithm and provide a rigorous theoretical analysis that proves finite-time convergence guarantees for PRQ under linear function approximation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
