We study reinforcement learning in infinite-horizon discounted Markov decision processes with continuous state spaces, where data are generated online from a single trajectory under a Markovian behavior policy. To avoid maintaining an infinite-dimensional, function-valued estimate, we propose the novel Q-Measure-Learning, which learns a signed empirical measure supported on visited state-action pairs and reconstructs an action-value estimate via kernel integration. The method jointly estimates the stationary distribution of the behavior chain and the Q-measure through coupled stochastic approximation, leading to an efficient weight-based implementation with $O(n)$ memory and $O(n)$ computation cost per iteration. Under uniform ergodicity of the behavior chain, we prove almost sure sup-norm convergence of the induced Q-function to the fixed point of a kernel-smoothed Bellman operator. We also bound the approximation error between this limit and the optimal $Q^*$ as a function of the kernel bandwidth. To assess the performance of our proposed algorithm, we conduct RL experiments in a two-item inventory control setting.
Relaxed Actor-Critic with Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems
University of Science and Technology Beijing(北京科技大学);Tsinghua University(清华大学);University of California Berkeley(加州大学伯克利分校);Beijing Idriverplus Technology Co., Ltd.(北京智行者科技有限公司)
Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation
University of Maryland(马里兰大学);Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico(高等理工学院);Universidade de Lisboa(里斯本大学);Hong Kong Baptist University(香港浸会大学);Cornell University(康奈尔大学)
On the Convergence of Continuous Constrained Optimization for Structure Learning
Carnegie Mellon University(卡内基梅隆大学);Mila, Université de Montréal(蒙特利尔大学米拉研究所);DeepMind(深度思维);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
New Q-Newton's method meets Backtracking line search: good convergence guarantee, saddle points avoidance, quadratic rate of convergence, and easy implementation