The cross-entropy method (CE) developed by R. Rubinstein is an elegant practical principle for simulating rare events. The method approximates the probability of the rare event by means of a family of probabilistic models. The method has been extended to optimization, by considering an optimal event as a rare event. CE works rather good when dealing with deterministic function optimization. Now, it appears that two conditions are needed for a good convergence of the method. First, it is necessary to have a family of models sufficiently flexible for discriminating the optimal events. Indirectly, it appears also that the function to be optimized should be deterministic. The purpose of this paper is to consider the case of partially discriminating model family, and of stochastic functions. It will be shown on simple examples that the CE could fail when relaxing these hypotheses. Alternative improvements of the CE method are investigated and compared on random examples in order to handle this issue.
Entropy Regularization for Mean Field Games with Learning
University of California, Berkeley（加州大学伯克利分校）；Amazon Research（亚马逊研究院）；University of Southern California（南加州大学）；University of Oxford（牛津大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Oxford-Man Institute, University of Oxford（牛津大学牛津曼研究所）
Variational discretization and semi-smooth Newton methods; implementation, convergence and globalization in pde constrained optimization with control constraints
Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regime
University of Edinburgh（爱丁堡大学）；Imperial College London（帝国理工学院）；Vega Protocol（维加协议）；The Alan Turing Institute（艾伦·图灵研究所）
研究了无限时域、连续状态和动作空间以及熵正则化马尔可夫决策过程（MDP）下策略梯度的整体收敛。我们考虑一个softmax政策（一个隐藏层）神经网络近似的平均场制度。在相关的平均场概率测度中增加了额外的熵正则化，并在2-Wasserstein度量中研究了相应的梯度流。我们表明，目标函数是增加沿梯度流。此外，我们证明了，如果正则化的平均场措施是足够的，梯度流指数快速收敛到唯一的固定的解决方案，这是唯一的最大化的正则化MDP目标。最后，我们研究了沿梯度流的值函数关于正则化参数和初始条件的灵敏度。我们的结果依赖于对非线性Fokker-Planck-Kolmogorov方程的仔细分析，并扩展了Mei et al. 2020和Agarwal et al. 2020的开创性工作，这些工作量化了表格设置中熵正则化MDP的政策梯度的整体收敛率。
