用于无模型策略梯度平均场控制的随机化传输映射

中文摘要 AI 辅助

我们开发了一种用于离散时间平均场控制（MFC）的无模型策略梯度方法。在MFC中，策略既通过受控动力学又通过群体分布影响目标函数。标准REINFORCE估计器捕捉到第一种效应，但未捕捉到第二种效应。我们引入了Transport REINFORCE，一种基于传输映射的方法，该方法通过对群体分布的合适变换进行扰动来估计缺失的平均场贡献。该方法适用于有限和连续状态空间。在有限状态空间中，我们通过当前群体权重与随机权重的凸组合，直接在概率单纯形上扰动群体分布。在连续状态空间中，我们将群体分布投影到高斯混合流形上，然后通过传输映射对其进行随机化，以确保扰动后的分布仍在该流形内。我们证明了当扰动消失时，扰动后的目标函数和梯度的一致性，并推导了所得基于样本的梯度估计器的偏差和均方误差界。在多个MFC基准上的数值实验表明，Transport REINFORCE优于标准REINFORCE。

英文摘要

We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.

发表机构

• ENS Paris-Saclay（巴黎萨克雷高等师范学校）
• École Polytechnique（巴黎综合理工学院）

机构由 AI 辅助整理，请以论文原文为准。