Model-based planners for partially observable problems must accommodate both model uncertainty during planning and goal uncertainty during objective inference. However, model-based planners may be brittle under these types of uncertainty because they rely on an exact model and tend to commit to a single optimal behavior. Inspired by results in the model-free setting, we propose an entropy-regularized model-based planner for partially observable problems. Entropy regularization promotes policy robustness for planning and objective inference by encouraging policies to be no more committed to a single action than necessary. We evaluate the robustness and objective inference performance of entropy-regularized policies in three problem domains. Our results show that entropy-regularized policies outperform non-entropy-regularized baselines in terms of higher expected returns under modeling errors and higher accuracy during objective inference.
Robust Regularized Policy Iteration under Transition Uncertainty
College of Computer Science and Technology, Zhejiang University, Hangzhou, China（浙江大学计算机科学与技术学院）；School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi'an, China（西北工业大学人工智能、光学与电子学院（iOPEN））；School of Software Technology, Zhejiang University, Hangzhou, China（浙江大学软件技术学院）；School of Software Engineering, Xi'an Jiaotong University, Xi'an, China（西安交通大学软件工程学院）；School of Systems Science and Engineering, Sun Yat-sen University, Guangzhou, China（中山大学系统科学与工程学院）
Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise
Department of Mathematics, University of California, Berkeley（加州大学伯克利分校数学系）；College of Computing, Data Science, and Society, University of California, Berkeley（加州大学伯克利分校计算机科学与数据科学学院）；Department of Applied Mathematics, The Hong Kong Polytechnic University（香港理工大学应用数学系）
Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regime
University of Edinburgh（爱丁堡大学）；Imperial College London（帝国理工学院）；Vega Protocol（维加协议）；The Alan Turing Institute（艾伦·图灵研究所）
研究了无限时域、连续状态和动作空间以及熵正则化马尔可夫决策过程（MDP）下策略梯度的整体收敛。我们考虑一个softmax政策（一个隐藏层）神经网络近似的平均场制度。在相关的平均场概率测度中增加了额外的熵正则化，并在2-Wasserstein度量中研究了相应的梯度流。我们表明，目标函数是增加沿梯度流。此外，我们证明了，如果正则化的平均场措施是足够的，梯度流指数快速收敛到唯一的固定的解决方案，这是唯一的最大化的正则化MDP目标。最后，我们研究了沿梯度流的值函数关于正则化参数和初始条件的灵敏度。我们的结果依赖于对非线性Fokker-Planck-Kolmogorov方程的仔细分析，并扩展了Mei et al. 2020和Agarwal et al. 2020的开创性工作，这些工作量化了表格设置中熵正则化MDP的政策梯度的整体收敛率。
KL-Entropy-Regularized RL with a Generative Model is Minimax Optimal
University of Alberta（阿尔伯塔大学）；Peking University（北京大学）；Google Research（谷歌研究院）；Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）；IECL（洛林计算机科学及其应用实验室）；University of Tokyo（东京大学）；DeepMind（深度思维）；Otto von Guericke University Magdeburg（奥托·冯·格里克马格德堡大学）
Applied and Theoretical Aspects of Robot Intelligence (ATARI) Lab（应用与理论机器人智能实验室）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）；Technical University of Munich（慕尼黑工业大学）
