Recently, the impressive empirical success of policy gradient (PG) methods has catalyzed the development of their theoretical foundations. Despite the huge efforts directed at the design of efficient stochastic PG-type algorithms, the understanding of their convergence to a globally optimal policy is still limited. In this work, we develop improved global convergence guarantees for a general class of Fisher-non-degenerate parameterized policies which allows to address the case of continuous state action spaces. First, we propose a Normalized Policy Gradient method with Implicit Gradient Transport (N-PG-IGT) and derive a $\tilde{\mathcal{O}}(\varepsilon^{-2.5})$ sample complexity of this method for finding a global $\varepsilon$-optimal policy. Improving over the previously known $\tilde{\mathcal{O}}(\varepsilon^{-3})$ complexity, this algorithm does not require the use of importance sampling or second-order information and samples only one trajectory per iteration. Second, we further improve this complexity to $\tilde{ \mathcal{\mathcal{O}} }(\varepsilon^{-2})$ by considering a Hessian-Aided Recursive Policy Gradient ((N)-HARPG) algorithm enhanced with a correction based on a Hessian-vector product. Interestingly, both algorithms are $(i)$ simple and easy to implement: single-loop, do not require large batches of trajectories and sample at most two trajectories per iteration; $(ii)$ computationally and memory efficient: they do not require expensive subroutines at each iteration and can be implemented with memory linear in the dimension of parameters.
Using machine learning to inform harvest control rule design in complex fishery settings
Eric and Wendy Schmidt Center for Data Science and Environment University of California, Berkeley（埃里克和wendy Schmidt数据科学与环境中心加州大学伯克利分校）；Department of Environmental Science, Policy, and Management University of California, Berkeley（环境科学、政策与管理系加州大学伯克利分校）；Institute for the Oceans and Fisheries, University of British Columbia（海洋与渔业研究所不列颠哥伦比亚大学）；Quantitative Fisheries Center, Department of Fisheries and Wildlife, Michigan State University（渔业定量中心渔业与野生动物系密歇根州立大学）
Center for Advanced Intelligence Project, RIKEN AIP, Japan（日本RIKEN AIP高级智能项目中心）；School of Mathematics, University of Edinburgh, UK（爱丁堡大学数学学院）；School of Mathematics, University of Edinburgh, UK, The Alan Turing Institute, UK（爱丁堡大学数学学院、英国艾伦·图灵研究所）
Fisher Decorator: Refining Flow Policy via A Local Transport Map
University College London（伦敦大学学院）；Shanghai Jiao Tong University（上海交通大学）；King’s College London（伦敦国王学院）；Peking University（北京大学）；Shanghai University of Finance and Economics（上海财经大学）；Imperial College London（伦敦帝国学院）
Achieving $\tilde{\mathcal{O}}(1/N)$ Optimality Gap in Restless Bandits through Gaussian Approximation
Electrical Engineering and Computer Science Department, University of Michigan, Ann Arbor（密歇根大学电气工程与计算机科学系）；Computer Science Department, Carnegie Mellon University（卡内基梅隆大学计算机科学系）
Synthesising Reinforcement Learning Policies through Set-Valued Inductive Rule Learning
Vrije Universiteit Brussel（布鲁塞尔自由大学（荷兰语））；Université Libre de Bruxelles（布鲁塞尔自由大学（法语））；Delft University of Technology（代尔夫特理工大学）；Leiden Institute of Advanced Computer Science（莱顿大学高级计算机科学研究所）
Optimal Control of Fluid Restless Multi-armed Bandits: A Machine Learning Approach
Sloan School of Management, Massachusetts Institute of Technology（麻省理工学院斯隆管理学院）；Operations Research Center, Massachusetts Institute of Technology（麻省理工学院运筹学中心）；Departamento de Estadı́stica, Universidad Carlos III de Madrid（马德里卡斯蒂利亚-拉曼查大学统计系）
