Implicit differentiation of Lasso-type models for hyperparameter optimization
Université Paris-Saclay（巴黎-萨克雷大学）；Inria（法国国家信息与自动化研究所）；CEA（法国原子能和替代能源委员会）；Institut Mathématique de Bourgogne（勃艮第数学研究所）；Université de Bourgogne（勃艮第大学）；Google Research（谷歌研究院）；CNRS（法国国家科学研究中心）；IMAG（蒙彼利埃亚历山大·格罗滕迪克研究所）；Université de Montpellier（蒙彼利埃大学）
Quentin Bertrand, Quentin Klopfenstein, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, Joseph Salmon
Setting regularization parameters for Lasso-type estimators is notoriously difficult, though crucial in practice. The most popular hyperparameter optimization approach is grid-search using held-out validation data. Grid-search however requires to choose a predefined grid for each parameter, which scales exponentially in the number of parameters. Another approach is to cast hyperparameter optimization as a bi-level optimization problem, one can solve by gradient descent. The key challenge for these methods is the estimation of the gradient with respect to the hyperparameters. Computing this gradient via forward or backward automatic differentiation is possible yet usually suffers from high memory consumption. Alternatively implicit differentiation typically involves solving a linear system which can be prohibitive and numerically unstable in high dimension. In addition, implicit differentiation usually assumes smooth loss functions, which is not the case for Lasso-type problems. This work introduces an efficient implicit differentiation algorithm, without matrix inversion, tailored for Lasso-type problems. Our approach scales to high-dimensional data by leveraging the sparsity of the solutions. Experiments demonstrate that the proposed method outperforms a large number of standard methods to optimize the error on held-out data, or the Stein Unbiased Risk Estimator (SURE).
Université Paris-Saclay（巴黎萨克雷大学）；Inria（法国国家信息与自动化研究所）；CEA（法国原子能和替代能源委员会）；Université de Bourgogne（勃艮第大学）；Institut Mathématique de Bourgogne（勃艮第数学研究所）；Università degli Studi di Genova（热那亚大学）；Google Research（谷歌研究院）；CNRS（法国国家科学研究中心）；Université de Montpellier（蒙彼利埃大学）
Finding the optimal hyperparameters of a model can be cast as a bilevel optimization problem, typically solved using zero-order techniques. In this work we study first-order methods when the inner optimization problem is convex but non-smooth. We show that the forward-mode differentiation of proximal gradient descent and proximal coordinate descent yield sequences of Jacobians converging toward the exact Jacobian. Using implicit differentiation, we show it is possible to leverage the non-smoothness of the inner problem to speed up the computation. Finally, we provide a bound on the error made …
There is a clear need for efficient algorithms to tune hyperparameters for statistical learning schemes, since the commonly applied search methods (such as grid search with N-fold cross-validation) are inefficient and/or approximate. Previously existing algorithms that efficiently search for hyperparameters relying on the smoothness of the cost function cannot be applied in problems such as Lasso regression. In this contribution, we develop a hyperparameter optimization method that relies on the structure of proximal gradient methods and does not require a smooth cost function. Such a method i…
Provably Data-driven Multiple Hyper-parameter Tuning with Structured Loss Function
Université Grenoble Alpes, LJK, CNRS, Grenoble INP（格拉诺布尔大学，LJK，CNRS，格拉诺布尔INP）；Carnegie Mellon University, Machine Learning Department（卡内基梅隆大学，机器学习系）；Chinese University of Hong Kong, Department of Systems Engineering and Engineering Management（香港中文大学，系统工程与工程管理系）
Shandong University（山东大学）；University of Geneva（日内瓦大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters pre…
The adaptive lasso refers to a class of methods that use weighted versions of the $L_1$-norm penalty, with weights derived from an initial estimate of the parameter vector to be estimated. Irrespective of the method chosen to compute this initial estimate, the performance of the adaptive lasso critically depends on the value of a hyperparameter, which controls the magnitude of the weighted $L_1$-norm in the penalized criterion. As for other machine learning methods, cross-validation is very popular for the calibration of the adaptive lasso, that this for the selection of a data-driven optimal …
In high dimensional sparse regression, pivotal estimators are estimators for which the optimal regularization parameter is independent of the noise level. The canonical pivotal estimator is the square-root Lasso, formulated along with its derivatives as a "non-smooth + non-smooth" optimization problem. Modern techniques to solve these include smoothing the datafitting term, to benefit from fast efficient proximal algorithms. In this work we show minimax sup-norm convergence rates for non smoothed and smoothed, single task and multitask square-root Lasso-type estimators. Thanks to our theoretic…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）
We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jo…
Hyperparameters tuning and model selection are important steps in machine learning. Unfortunately, classical hyperparameter calibration and model selection procedures are sensitive to outliers and heavy-tailed data. In this work, we construct a selection procedure which can be seen as a robust alternative to cross-validation and is based on a median-of-means principle. Using this procedure, we also build an ensemble method which, trained with algorithms and corrupted heavy-tailed data, selects an algorithm, trains it with a large uncorrupted subsample and automatically tune its hyperparameters…
Bilevel optimization has arisen as a powerful tool for many machine learning problems such as meta-learning, hyperparameter optimization, and reinforcement learning. In this paper, we investigate the nonconvex-strongly-convex bilevel optimization problem. For deterministic bilevel optimization, we provide a comprehensive convergence rate analysis for two popular algorithms respectively based on approximate implicit differentiation (AID) and iterative differentiation (ITD). For the AID-based method, we orderwisely improve the previous convergence rate analysis due to a more practical parameter …
University of Cambridge（剑桥大学）；Alan Turing Institute（艾伦·图灵研究所）
Machine learning training methods depend plentifully and intricately on hyperparameters, motivating automated strategies for their optimisation. Many existing algorithms restart training for each new hyperparameter choice, at considerable computational cost. Some hypergradient-based one-pass methods exist, but these either cannot be applied to arbitrary optimiser hyperparameters (such as learning rates and momenta) or take several times longer to train than their base models. We extend these existing methods to develop an approximate hypergradient-based hyperparameter optimiser which is applic…
Distribution-dependent Generalization Bounds for Tuning Linear Regression Across Tasks
School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）；Machine Learning Department, Carnegie Mellon University（卡内基梅隆大学机器学习系）；Toyota Technological Institute at Chicago（芝加哥丰田技术学院）；Northwestern University（西北大学）
Generalization is a central problem in Machine Learning. Indeed most prediction methods require careful calibration of hyperparameters usually carried out on a hold-out \textit{validation} dataset to achieve generalization. The main goal of this paper is to introduce a novel approach to achieve generalization without any data splitting, which is based on a new risk measure which directly quantifies a model's tendency to overfit. To fully understand the intuition and advantages of this new approach, we illustrate it in the simple linear regression model ($Y=Xβ+ξ$) where we develop a new criteri…
Exploring the Generalization Capabilities of AID-based Bi-level Optimization
Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Tencent（腾讯）；Tencent AI Lab（腾讯AI Lab）
University of Queensland（昆士兰大学）；University of Caen（卡昂大学）；La Trobe University（拉筹伯大学）
Mixtures-of-Experts models and their maximum likelihood estimation (MLE) via the EM algorithm have been thoroughly studied in the statistics and machine learning literature. They are subject of a growing investigation in the context of modeling with high-dimensional predictors with regularized MLE. We examine MoE with Gaussian gating network, for clustering and regression, and propose an $\ell_1$-regularized MLE to encourage sparse models and deal with the high-dimensional setting. We develop an EM-Lasso algorithm to perform parameter estimation and utilize a BIC-like criterion to select the m…
Gradient-based hyperparameter optimization has earned a widespread popularity in the context of few-shot meta-learning, but remains broadly impractical for tasks with long horizons (many gradient steps), due to memory scaling and gradient degradation issues. A common workaround is to learn hyperparameters online, but this introduces greediness which comes with a significant performance drop. We propose forward-mode differentiation with sharing (FDS), a simple and efficient algorithm which tackles memory scaling issues with forward-mode differentiation, and gradient degradation issues by sharin…
Bilevel optimization has become a powerful framework in various machine learning applications including meta-learning, hyperparameter optimization, and network architecture search. There are generally two classes of bilevel optimization formulations for machine learning: 1) problem-based bilevel optimization, whose inner-level problem is formulated as finding a minimizer of a given loss function; and 2) algorithm-based bilevel optimization, whose inner-level solution is an output of a fixed algorithm. For the first class, two popular types of gradient-based algorithms have been proposed for hy…
Learning Distributionally Robust First-Order Methods for Convex Optimization
Department of Operations Research and Financial Engineering, Princeton University（普林斯顿大学运筹学与金融工程系）；Research Institute of Mathematics, Seoul National University（首尔国立大学数学研究所）
University of Žilina（日利纳大学）；Durham University（杜伦大学）；ElaadNL
Here, we develop a data-centric approach enabling to analyse which activities, function, and characteristics of the environment surrounding the slow charging infrastructure impact the distribution of the electricity consumed at slow charging infrastructure. To gain a basic insight, we analysed the probabilistic distribution of energy consumption and its relation to indicators characterizing charging events. We collected geospatial datasets and utilizing statistical methods for data pre-processing, we prepared features modelling the spatial context in which the charging infrastructure operates.…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Hyperparameter optimization of neural networks can be elegantly formulated as a bilevel optimization problem. While research on bilevel optimization of neural networks has been dominated by implicit differentiation and unrolling, hypernetworks such as Self-Tuning Networks (STNs) have recently gained traction due to their ability to amortize the optimization of the inner objective. In this paper, we diagnose several subtle pathologies in the training of STNs. Based on these observations, we propose the $Δ$-STN, an improved hypernetwork architecture which stabilizes training and optimizes hyperp…
Variational Garrote for Statistical Physics-based Sparse and Robust Variable Selection
Department of Physics Education, Seoul National University, Seoul 08826, Korea（物理教育系，首尔国立大学）；Laboratory of Biological Modeling, National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health, Bethesda, Maryland 20892, USA（生物建模实验室，国家糖尿病与消化系统疾病研究所，美国国家卫生研究院，贝塞斯达，马里兰州）；Center for Theoretical Physics and Artificial Intelligence Institute, Seoul National University, Seoul 08826, Korea（理论物理与人工智能研究所，首尔国立大学）；School of Computational Sciences, Korea Institute for Advanced Study, Seoul 02455, Korea（计算科学学院，韩国高级研究院）
We investigate the choice of tuning parameters for a Bayesian multi-level group lasso model developed for the joint analysis of neuroimaging and genetic data. The regression model we consider relates multivariate phenotypes consisting of brain summary measures (volumetric and cortical thickness values) to single nucleotide polymorphism (SNPs) data and imposes penalization at two nested levels, the first corresponding to genes and the second corresponding to SNPs. Associated with each level in the penalty is a tuning parameter which corresponds to a hyperparameter in the hierarchical Bayesian f…
Solving a bilevel optimization problem is at the core of several machine learning problems such as hyperparameter tuning, data denoising, meta- and few-shot learning, and training-data poisoning. Different from simultaneous or multi-objective optimization, the steepest descent direction for minimizing the upper-level cost in a bilevel problem requires the inverse of the Hessian of the lower-level cost. In this work, we propose a novel algorithm for solving bilevel optimization problems based on the classical penalty function approach. Our method avoids computing the Hessian inverse and can han…
