In this article we present a generalised view on Path Integral Control (PIC) methods. PIC refers to a particular class of policy search methods that are closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems. This class is unique in the sense that it can be solved explicitly to yield a formal optimal state trajectory distribution. In this contribution we first review the PIC theory and discuss related algorithms tailored to policy search in general. We are able to identify a generic design strategy that relies on the existence of an optimal state trajectory distribution and finds a parametric policy by minimizing the cross entropy between the optimal and a state trajectory distribution parametrized through its policy. Inspired by this observation we then aim to formulate a SOC problem that shares traits with the LSOC setting yet that covers a less restrictive class of problem formulations. We refer to this SOC problem as Entropy Regularized Trajectory Optimization. The problem is closely related to the Entropy Regularized Stochastic Optimal Control setting which is lately often addressed by the Reinforcement Learning (RL) community. We analyse the theoretical convergence behaviour of the theoretical state trajectory distribution sequence and draw connections with stochastic search methods tailored to classic optimization problems. Finally we derive explicit updates and compare the implied Entropy Regularized PIC with earlier work in the context of both PIC and RL for derivative-free trajectory optimization.
Fast convergence rates and trajectory convergence of a Tikhonov regularized inertial primal\mbox{-}dual dynamical system with time scaling and vanishing damping
Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Smooth or Separable? Sparse Block Acceleration for Entropy-Regularized Linear Programs
Moscow Independent Research Institute of Artificial Intelligence（莫斯科独立人工智能研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Lomonosov Moscow State University（莫斯科罗蒙诺索夫国立大学）；Innopolis University（因诺波利斯大学）
Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regime
University of Edinburgh（爱丁堡大学）；Imperial College London（帝国理工学院）；Vega Protocol（维加协议）；The Alan Turing Institute（艾伦·图灵研究所）
研究了无限时域、连续状态和动作空间以及熵正则化马尔可夫决策过程（MDP）下策略梯度的整体收敛。我们考虑一个softmax政策（一个隐藏层）神经网络近似的平均场制度。在相关的平均场概率测度中增加了额外的熵正则化，并在2-Wasserstein度量中研究了相应的梯度流。我们表明，目标函数是增加沿梯度流。此外，我们证明了，如果正则化的平均场措施是足够的，梯度流指数快速收敛到唯一的固定的解决方案，这是唯一的最大化的正则化MDP目标。最后，我们研究了沿梯度流的值函数关于正则化参数和初始条件的灵敏度。我们的结果依赖于对非线性Fokker-Planck-Kolmogorov方程的仔细分析，并扩展了Mei et al. 2020和Agarwal et al. 2020的开创性工作，这些工作量化了表格设置中熵正则化MDP的政策梯度的整体收敛率。
Tikhonov regularized second-order plus first-order primal-dual dynamical systems with asymptotically vanishing damping for linear equality constrained convex optimization problems
Equilibrium for Time-inconsistent Mean Field Games: A Systematic Analysis by Entropy Regularization
University of Michigan（密歇根大学）；Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究院）；The Hong Kong Polytechnic University（香港理工大学）；Tsinghua University（清华大学）
Relaxed Control with Entropy Regularization for Itô Stochastic Systems with Input Delay
School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；Department of Applied Mathematics, The Hong Kong Polytechnic University（香港理工大学应用数学系）；School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）
