arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.36758math.OCcs.LG
收藏

用于扩散过程的频率一致对数后悔强化学习

Frequency-Uniform Log-Regret Reinforcement Learning for Diffusions

Stanford University(斯坦福大学)

Jose Blanchet, Lutong Hao, Renyuan Xu

AI总结:

针对高频控制问题离散化导致的后悔保证恶化,本文提出用于扩散过程的频率一致对数后悔强化学习,证明了与时间步长无关的后悔界,并验证了CD-FQ算法的有效性。

摘要:

高频控制问题常通过时间离散化研究,但细化时间步长δ会增大离散时域H=T/δ,即使物理时域T固定,也可能导致标准有限时域后悔保证恶化。本文研究受控扩散过程欧拉离散化的回合式基于值的强化学习,性能以K个回合相对于对应欧拉模型最优策略的累积后悔衡量。在值函数梯度、贝尔曼残差导数及拟合评论者的稳定性与正则性条件下,证明了与δ无关的常数的O(log K)后悔界。对于漂移为未知参数线性函数、扩散协方差为未知常数的动力学,验证了约束型基于模型的受控扩散拟合Q算法(CD-FQ)的这些条件的局域版本,得到固定有限T下高概率O(log K log log K)后悔,且常数与δ无关。

英文摘要:

High-frequency control problems are often studied through time discretization, but refining the timestep $δ$ increases the discrete horizon $H=T/δ$ and can make standard finite-horizon regret guarantees deteriorate even when the physical horizon $T$ is fixed. We study episodic value-based reinforcement learning for Euler discretizations of controlled diffusions. The performance is measured as the cumulative regret over $K$ episodes relative to the optimal policy of the corresponding Euler model. Under stability and regularity conditions on value function gradients, Bellman residual derivatives, and fitted critics, we prove an $\mathcal{O}(\log K)$ regret bound with constants uniform in $δ$. For dynamics with drift linear in unknown parameters and unknown constant diffusion covariance, we verify localized versions of these conditions for a constrained, model-based controlled-diffusion fitted-$Q$ algorithm (CD-FQ) and obtain high-probability $\mathcal{O}(\log K\log\log K)$ regret for fixed and finite $T$, again with constants uniform in $δ$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑