How much data is sufficient to learn high-performing algorithms? Generalization guarantees for data-driven algorithm design
Carnegie Mellon University(卡内基梅隆大学);University of Texas at El Paso(德克萨斯大学埃尔帕索分校);University of Pennsylvania(宾夕法尼亚大学);Ocean Genomics, Inc.(海洋基因组学公司);Optimized Markets, Inc.(优化市场公司);Strategic Machine, Inc.(战略机器公司);Strategy Robot, Inc.(战略机器人公司)
Maria-Florina Balcan, Dan DeBlasio, Travis Dick, Carl Kingsford, Tuomas Sandholm, Ellen Vitercik
Algorithms often have tunable parameters that impact performance metrics such as runtime and solution quality. For many algorithms used in practice, no parameter settings admit meaningful worst-case bounds, so the parameters are made available for the user to tune. Alternatively, parameters may be tuned implicitly within the proof of a worst-case approximation ratio or runtime bound. Worst-case instances, however, may be rare or nonexistent in practice. A growing body of research has demonstrated that data-driven algorithm design can lead to significant improvements in performance. This approach uses a training set of problem instances sampled from an unknown, application-specific distribution and returns a parameter setting with strong average performance on the training set.
We provide a broadly applicable theory for deriving generalization guarantees that bound the difference between the algorithm's average performance over the training set and its expected performance. Our results apply no matter how the parameters are tuned, be it via an automated or manual approach. The challenge is that for many types of algorithms, performance is a volatile function of the parameters: slightly perturbing the parameters can cause large changes in behavior. Prior research has proved generalization bounds by employing case-by-case analyses of greedy algorithms, clustering algorithms, integer programming algorithms, and selling mechanisms. We uncover a unifying structure which we use to prove extremely general guarantees, yet we recover the bounds from prior research. Our guarantees apply whenever an algorithm's performance is a piecewise-constant, -linear, or -- more generally -- piecewise-structured function of its parameters. Our theory also implies novel bounds for voting mechanisms and dynamic programming algorithms from computational biology.
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
最近一系列关于深度学习的研究集中在极端过度参数化的设置上,并表明当网络宽度大于训练样本大小$n$和目标误差$ε^{-1}$的倒数的高次多项式时,通过(随机)梯度下降学习的深度神经网络具有良好的优化和泛化保证。最近,研究表明,在对训练数据的某些裕度假设下,多对数宽度条件足以使双层ReLU网络收敛和泛化(Ji and Telgarsky,2019)。然而,深度神经网络是否可以通过这种温和的过度参数化来学习仍然是一个悬而未决的问题。在这项工作中,我们肯定地回答了这个问题,并为通过(随机)梯度下降训练的深度ReLU网络建立了更清晰的学习保证。具体而言,在一定的假设下,在以前的工作中,我们的优化和推广保证持有$n$和$ε^{-1}$的网络宽度多对数。我们的研究结果将过度参数化深度神经网络的研究推向了更实用的设置。
Algorithm Design and Stronger Guarantees for the Improving Multi-Armed Bandits Problem
Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);University of Chicago(芝加哥大学);IDEAL Institute, Toyota Technological Institute at Chicago(IDEAL研究所,芝加哥丰田技术研究所)
OPTIMUS-Prime: Minimal and Sufficient Concept Explanations for Deep Vision Models
Université de Lorraine(洛林大学);CentraleSupélec Loria(中央超导Loria);CNRS(国家科学研究中心);Metz, France(法国梅斯);Université de technologie de Compiègne UMR CNRS 7253 Heudiasyc(图卢兹技术大学UMR CNRS 7253 Heudiasyc);France(法国)
Learning-Augmented Algorithms: Guarantees, Construction Mechanisms, and System-Level Implications
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院)