arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.03523cs.LGmath.OC
收藏

Q度量学习用于连续状态强化学习:高效的实现与收敛

Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence

Shengbo Wang

AI总结:

本文提出Q度量学习方法,用于连续状态强化学习,通过核积分高效估计动作值函数,并证明其在核带宽下的收敛性。

摘要:

我们研究了在无限时间折扣马尔可夫决策过程中,具有连续状态空间的强化学习问题,其中数据是在线从单条轨迹下由马尔可夫行为策略生成的。为了避免维护一个无限维、函数值估计的估计,我们提出了新颖的Q度量学习方法,该方法学习一个支持在已访问状态-动作对上的有号经验度量,并通过核积分重建动作值估计。该方法通过耦合的随机逼近联合估计行为链的平稳分布和Q度量,导致具有O(n)内存和每迭代O(n)计算成本的高效的权重基于实现。在行为链的均匀指数性下,我们证明了所诱导的Q函数在核平滑贝尔曼算子的固定点处的几乎必然上范数收敛。我们还界定了该极限与最优Q*之间的近似误差作为核带宽函数的函数。为了评估我们所提算法的性能,我们在两个物品库存控制设置中进行了强化学习实验。

英文摘要:

We study reinforcement learning in infinite-horizon discounted Markov decision processes with continuous state spaces, where data are generated online from a single trajectory under a Markovian behavior policy. To avoid maintaining an infinite-dimensional, function-valued estimate, we propose the novel Q-Measure-Learning, which learns a signed empirical measure supported on visited state-action pairs and reconstructs an action-value estimate via kernel integration. The method jointly estimates the stationary distribution of the behavior chain and the Q-measure through coupled stochastic approximation, leading to an efficient weight-based implementation with $O(n)$ memory and $O(n)$ computation cost per iteration. Under uniform ergodicity of the behavior chain, we prove almost sure sup-norm convergence of the induced Q-function to the fixed point of a kernel-smoothed Bellman operator. We also bound the approximation error between this limit and the optimal $Q^*$ as a function of the kernel bandwidth. To assess the performance of our proposed algorithm, we conduct RL experiments in a two-item inventory control setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑