arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.27372math.OCcs.LGcs.MA
收藏

连续时间q学习用于具有公共噪声的均场控制,第一部分:理论基础

Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations

LaMME, Université Évry Paris-Saclay(Évry巴黎萨克雷大学LaMME研究中心);Department of Applied Mathematics, The Hong Kong Polytechnic University(香港理工大学应用数学系);Department of Industrial Engineering and Operations Research, Columbia University(哥伦比亚大学工业工程与运筹学系)

Zhenjie Ren, Xiaoli Wei, Xiang Yu, Xun Yu Zhou

AI总结:

本文研究了熵正则化均场控制中连续时间q函数的特性,通过离散采样动作证明价值函数收敛,并推导了探索性HJB方程,建立了最优策略迭代的存在性和唯一性,最后在一般线性二次设定中显式刻画最优策略。

摘要:

本文研究了熵正则化均场控制中连续时间q函数的特性,该q函数由Jia和Zhou(2023)在单体模型中提出。我们首先证明,在离散采样动作下,探索性表述中的价值函数在时间网格细化时收敛到放松控制表述中的价值函数。利用放松控制表述,我们推导了探索性哈密顿-雅可比-贝尔曼(HJB)方程,其中受控公共噪声导致政策产生额外的非线性功能,使策略迭代复杂化。在一定凹凸条件下,我们通过使用关于策略的偏线性泛函导数的一阶条件,建立了最优一步策略迭代的存在性和唯一性。每次迭代中的策略改进通过与策略空间上的熵正则化优化问题相关联来验证。在均场设置中,我们引入了定义在状态分布和策略上的集成q函数(Iq函数),并证明最优策略是Iq函数argmax算子的双层固定点。最后,我们提供了在一般线性二次(LQ)设定中最优策略的显式刻画,作为高斯分布。

英文摘要:

This paper investigates the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise, coined as q-function by Jia and Zhou (2023) in the single agent's model. We first show that, under discretely sampled actions, the value function in the exploratory formulation converges to the one in the relaxed control formulation as the time grid refines. Leveraging the relaxed control formulation, we derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy, rendering the policy iteration intricate. Under certain concavity condition, we establish the existence and uniqueness of the optimal one-step policy iteration via a first-order condition using the partial linear functional derivative with respect to policy. The policy improvement at each iteration is verified by relating to an entropy-regularized optimization problem over the space of policies. In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function. Finally, we provide the explicit characterization of an optimal policy as a Gaussian distribution in the general linear-quadratic (LQ) setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑