This paper is concerned with the sample efficiency of reinforcement learning, assuming access to a generative model (or simulator). We first consider $γ$-discounted infinite-horizon Markov decision processes (MDPs) with state space $\mathcal{S}$ and action space $\mathcal{A}$. Despite a number of prior works tackling this problem, a complete picture of the trade-offs between sample complexity and statistical accuracy is yet to be determined. In particular, all prior results suffer from a severe sample size barrier, in the sense that their claimed statistical guarantees hold only when the sample size exceeds at least $\frac{|\mathcal{S}||\mathcal{A}|}{(1-γ)^2}$. The current paper overcomes this barrier by certifying the minimax optimality of two algorithms -- a perturbed model-based algorithm and a conservative model-based algorithm -- as soon as the sample size exceeds the order of $\frac{|\mathcal{S}||\mathcal{A}|}{1-γ}$ (modulo some log factor). Moving beyond infinite-horizon MDPs, we further study time-inhomogeneous finite-horizon MDPs, and prove that a plain model-based planning algorithm suffices to achieve minimax-optimal sample complexity given any target accuracy level. To the best of our knowledge, this work delivers the first minimax-optimal guarantees that accommodate the entire range of sample sizes (beyond which finding a meaningful policy is information theoretically infeasible).
Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Optimization
École normale supérieure(巴黎高等师范学院);CNRS(法国国家科学研究中心);PSL Research University(巴黎文理研究大学);INRIA(法国国家信息与自动化研究所);ENS(巴黎高等师范学院);MILA(蒙特利尔学习算法研究所);DIRO(蒙特利尔大学计算机科学与运筹学系);Université de Montréal(蒙特利尔大学)
Provably Safe Generative Sampling with Constricting Barrier Functions
Mechanical Engineering University of California, Riverside(机械工程 加州大学河滨分校);Electrical Engineering and Computer Science University of California, Irvine(电气工程与计算机科学 加州大学伊尔弗分校)
Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads
University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Ohio State University(俄亥俄州立大学);Chinese Academy of Sciences(中国科学院);University of California, Riverside(加州大学河滨分校);Microsoft Research(微软研究院)
Breaking the Tuning Barrier: Zero-Hyperparameters Yield Multi-Corner Analysis Via Learned Priors
School of Mathematical and Physical Science, University of Sheffield(谢菲尔德大学数学与物理科学学院);SZU–UoS Joint Centre for Innovation and Entrepreneurship, College of Mechatronics and Control Engineering, Shenzhen University(深大-乌兹别克斯坦联合创新与创业中心,机电控制工程学院,深圳大学);Nanjing University of Science and Technology(南京理工大学)
Breaking the Data Barrier in Learning Symbolic Computation: A Case Study on Variable Ordering Suggestion for Cylindrical Algebraic Decomposition
School of Mathematical Sciences, Jiangsu University(江苏大学数学科学学院);Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences(中国科学院重庆绿色智能技术研究所);Chongqing School, University of Chinese Academy of Sciences(中国科学院大学重庆学院)
Spontaneous Symmetry Breaking in Generative Diffusion Models
Jheronimus Academy of Data Science(杰罗尼穆斯数据科学学院);Tilburg University(蒂尔堡大学);Radboud University(拉德堡德大学);Donders Institute for Brain, Cognition and Behaviour(唐德斯大脑、认知与行为研究所)