arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.00716cs.LGstat.ML
收藏

冻结策略迭代:在确定性动态下线性$Q^π$可实现性下的计算高效强化学习

Frozen Policy Iteration: Computationally Efficient RL under Linear $Q^π$ Realizability for Deterministic Dynamics

School of EECS, Peking University(北京大学电子工程系);Department of Computer Science and Engineering, HKUST(香港科技大学计算机科学与工程系);CFCS and School of Computer Science, Peking University(北京大学计算机学院)

Yijing Ke, Zihan Zhang, Ruosong Wang

AI总结:

本文提出冻结策略迭代算法,在确定性动态下实现计算高效的强化学习,通过在线策略迭代和冻结策略技术,达到最优 regrets bound。

摘要:

我们研究了在线性$Q^π$可实现性假设下的计算和统计高效强化学习,其中任何策略的$Q$-函数在给定的状态-动作特征表示中是线性的。先前方法在此设定下要么计算上不可行,要么需要(局部)访问模拟器。在本文中,我们提出了一种计算高效的在线强化学习算法,名为冻结策略迭代,适用于具有随机初始状态、随机奖励和确定性转移的马尔可夫决策过程(MDPs)。我们的算法实现了$\widetilde{O}(\sqrt{d^2H^6T})$的 regrets bound,其中$d$是特征空间的维度,$H$是时间 horizon 长度,$T$是总episode数。我们的 regrets bound 对于线性(上下文)带宽来说是最优的,这是我们设定的一个特殊情况,其中$H=1$。现有的策略迭代算法在相同设定下严重依赖于通过访问模拟器反复采样相同状态,这在本文研究的具有随机初始状态的在线设置中不可行。相反,我们的新算法通过战略性地使用仅高置信度的部分轨迹数据并冻结已探索状态的策略,避免了这一限制,确保了算法在整个学习过程中所有使用的数据都保持有效 on-policy。我们进一步通过将方法扩展到均匀-PAC 设置和具有有界 eluder 维度的功能类,展示了该方法的通用性。

英文摘要:

We study computationally and statistically efficient reinforcement learning under the linear $Q^π$ realizability assumption, where any policy's $Q$-function is linear in a given state-action feature representation. Prior methods in this setting are either computationally intractable, or require (local) access to a simulator. In this paper, we propose a computationally efficient online RL algorithm, named Frozen Policy Iteration, under the linear $Q^π$ realizability setting that works for Markov Decision Processes (MDPs) with stochastic initial states, stochastic rewards and deterministic transitions. Our algorithm achieves a regret bound of $\widetilde{O}(\sqrt{d^2H^6T})$, where $d$ is the dimensionality of the feature space, $H$ is the horizon length, and $T$ is the total number of episodes. Our regret bound is optimal for linear (contextual) bandits which is a special case of our setting with $H = 1$. Existing policy iteration algorithms under the same setting heavily rely on repeatedly sampling the same state by access to the simulator, which is not implementable in the online setting with stochastic initial states studied in this paper. In contrast, our new algorithm circumvents this limitation by strategically using only high-confidence part of the trajectory data and freezing the policy for well-explored states, which ensures that all data used by our algorithm remains effectively on-policy during the whole course of learning. We further demonstrate the versatility of our approach by extending it to the Uniform-PAC setting and to function classes with bounded eluder dimension.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑