一种适用于通用函数近似强化学习的近最优低切换算法
A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation
AI总结：
针对通用函数近似强化学习的探索-利用困境，提出MQL-UCB算法，通过低切换策略、单调价值结构与方差加权回归，实现近最优悔值与切换成本，为高效非线性Q学习设计提供支撑。
摘要：
探索-利用困境一直是复杂模型类下强化学习（RL）的核心挑战。本文提出了一种适用于通用函数近似RL的新算法——带置信上界的单调Q学习（Monotonic Q-Learning with Upper Confidence Bound, MQL-UCB）。该算法的核心设计包括：（1）可实现低切换成本的通用确定性策略切换策略；（2）函数类复杂度得到精细控制的单调价值函数结构；（3）可高效利用历史轨迹、数据效率极高的方差加权回归方案。当K足够大时，MQL-UCB可达到$\tilde{O}(d\sqrt{HK})$的极小极大最优悔值，以及$\tilde{O}(dH)$的近最优策略切换成本，其中d为函数类的eluder维度，H为规划时域，K为回合数。本研究为设计具备可证明样本效率与部署效率的非线性函数近似Q学习提供了思路。
英文摘要：
The exploration-exploitation dilemma has been a central challenge in reinforcement learning (RL) with complex model classes. In this paper, we propose a new algorithm, Monotonic Q-Learning with Upper Confidence Bound (MQL-UCB) for RL with general function approximation. Our key algorithmic design includes (1) a general deterministic policy-switching strategy that achieves low switching cost, (2) a monotonic value function structure with carefully controlled function class complexity, and (3) a variance-weighted regression scheme that exploits historical trajectories with high data efficiency. MQL-UCB achieves minimax optimal regret of $\tilde{O}(d\sqrt{HK})$ when $K$ is sufficiently large and near-optimal policy switching cost of $\tilde{O}(dH)$, with $d$ being the eluder dimension of the function class, $H$ being the planning horizon, and $K$ being the number of episodes. Our work sheds light on designing provably sample-efficient and deployment-efficient Q-learning with nonlinear function approximation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
