经典优化理论要求基于梯度的方法采用小步长才能收敛。然而，近期研究发现对传统观点提出了挑战：实验证明即使步长$η$超过$2/L$的阈值（其中$L$为全局光滑常数），梯度下降（GD）依然能够收敛，这一现象通常被称为稳定边缘（Edge of Stability, EoS）现象。学界普遍认为，具有次二次增长的目标函数在引发EoS现象中发挥着重要作用。\n本文针对回归任务中寻找线性插值器$β\in R^{d}$的问题给出了更全面的解答，其中损失函数为$l(\cdot)$，$β$可参数化为$β= w^2_{+} - w^2_{-}$。与此前认为次二次$l$是EoS必要条件的研究不同，本文的新发现表明：在适当条件下，即使$l$为二次函数，EoS现象依然会发生。本文通过实验与理论证据双重验证了这一结论的严谨性，证明梯度下降轨迹会以非渐近的方式收敛到线性插值器。此外，二次$l$下的模型（也被称为深度为2的对角线性网络）在EoS机制下的研究目前仍较为匮乏。本文的分析为采用大步长时对角线性网络的隐式偏差提供了新的见解，丰富了对更具实用性的模型中EoS现象的理解。
英文摘要：
Classical optimization theory requires a small step-size for gradient-based methods to converge. Nevertheless, recent findings challenge the traditional idea by empirically demonstrating Gradient Descent (GD) converges even when the step-size $η$ exceeds the threshold of $2/L$, where $L$ is the global smooth constant. This is usually known as the Edge of Stability (EoS) phenomenon. A widely held belief suggests that an objective function with subquadratic growth plays an important role in incurring EoS. In this paper, we provide a more comprehensive answer by considering the task of finding linear interpolator $β\in R^{d}$ for regression with loss function $l(\cdot)$, where $β$ admits parameterization as $β= w^2_{+} - w^2_{-}$. Contrary to the previous work that suggests a subquadratic $l$ is necessary for EoS, our novel finding reveals that EoS occurs even when $l$ is quadratic under proper conditions. This argument is made rigorous by both empirical and theoretical evidence, demonstrating the GD trajectory converges to a linear interpolator in a non-asymptotic way. Moreover, the model under quadratic $l$, also known as a depth-$2$ diagonal linear network, remains largely unexplored under the EoS regime. Our analysis then sheds some new light on the implicit bias of diagonal linear networks when a larger step-size is employed, enriching the understanding of EoS on more practical models.
Over-parameterization and adaptive methods have played a crucial role in the success of deep learning in the last decade. The widespread use of over-parameterization has forced us to rethink generalization by bringing forth new phenomena, such as implicit regularization of optimization algorithms and double descent with training progression. A series of recent works have started to shed light on these areas in the quest to understand -- why do neural networks generalize well? The setting of over-parameterized linear regression has provided key insights into understanding this mysterious behavi…
A fundamental principle of learning theory is that there is a trade-off between the complexity of a prediction rule and its ability to generalize. Modern machine learning models do not obey this paradigm: They produce an accurate prediction even with a perfect fit to the training set. We investigate over-parameterized linear regression models focusing on the minimum norm solution: This is the solution with the minimal norm that attains a perfect fit to the training set. We utilize the recently proposed predictive normalized maximum likelihood (pNML) learner which is the min-max regret solution…
We consider minimax-optimal designs for the prediction of individual parameters in random coefficient regression models. We focus on the minimax-criterion, which minimizes the "worst case" for the basic criterion with respect to the covariance matrix of random effects. We discuss particular models: linear and quadratic regression, in detail.
The paper addresses the problem of learning a regression model parameterized by a fixed-rank positive semidefinite matrix. The focus is on the nonlinear nature of the search space and on scalability to high-dimensional problems. The mathematical developments rely on the theory of gradient descent algorithms adapted to the Riemannian geometry that underlies the set of fixed-rank positive semidefinite matrices. In contrast with previous contributions in the literature, no restrictions are imposed on the range space of the learned matrix. The resulting algorithms maintain a linear complexity in t…
Sharp Generalization for Nonparametric Regression in Interpolation Space by Over-Parameterized Neural Networks Trained with Preconditioned Gradient Descent and Early Stopping
School of Computing and Augmented Intelligence（计算与增强智能学院）；Arizona State University（亚利桑那州立大学）；VecML Inc.（VecML公司）
We propose new model selection criteria based on generalized ridge estimators dominating the maximum likelihood estimator under the squared risk and the Kullback-Leibler risk in multivariate linear regression. Our model selection criteria have the following favorite properties: consistency, unbiasedness, uniformly minimum variance. Consistency is proven under an asymptotic structure $\frac{p}{n}\to c$ where $n$ is the sample size and $p$ is the parameter dimension of the response variables. In particular, our proposed class of estimators dominates the maximum likelihood estimator under the squ…
École Centrale Paris（巴黎中央理工学院）；Institut HyperCube（超立方体研究所）；Hopital Bichat-Claude Bernard（比沙-克洛德·贝尔纳医院）
Multivariate classification methods using explanatory and predictive models are necessary for characterizing subgroups of patients according to their risk profiles. Popular methods include logistic regression and classification trees with performances that vary according to the nature and the characteristics of the dataset. In the context of imported malaria, we aimed at classifying severity criteria based on a heterogeneous patient population. We investigated these approaches by implementing two different strategies: L1 logistic regression (L1LR) that models a single global solution and class…
We consider nonsynchronous sampling of parameterized stochastic regression models, which contain stochastic differential equations. Constructing a quasi-likelihood function, we prove that the quasi-maximum likelihood estimator and the Bayes type estimator are consistent and asymptotically mixed normal when the sampling frequency of the nonsynchronous data becomes large.
A Comparative Study of Model Selection Criteria for Symbolic Regression
Aarhus University（奥胡斯大学）；Heuristic and Evolutionary Algorithms Laboratory (HEAL)（启发式和进化算法实验室）；Federal University of ABC（巴西联邦大学ABC分校）；University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）；University of Modena（摩德纳大学）；Reggio Emilia Department of Physics, Informatics（雷吉奥艾米利亚物理与信息学系）
We consider a new criterion-based approach to model selection in linear regression. Properties of selection criteria based on p-values of a likelihood ratio statistic are studied for families of linear regression models. We prove that such procedures are consistent i.e. the minimal true model is chosen with probability tending to 1 even when the number of models under consideration slowly increases with a sample size. The simulation study indicates that introduced methods perform promisingly when compared with Akaike and Bayesian Information Criteria.
This work is substituted by the paper in arXiv:2011.14066. Stochastic gradient descent is the de facto algorithm for training deep neural networks (DNNs). Despite its popularity, it still requires fine tuning in order to achieve its best performance. This has led to the development of adaptive methods, that claim automatic hyper-parameter optimization. Recently, researchers have studied both algorithmic classes via toy examples: e.g., for over-parameterized linear regression, Wilson et. al. (2017) shows that, while SGD always converges to the minimum-norm solution, adaptive methods show no suc…
Taylor University（泰勒大学）；Oak Ridge National Laboratory（橡树岭国家实验室）
Variational training of parameterized quantum circuits (PQCs) underpins many workflows employed on near-term noisy intermediate scale quantum (NISQ) devices. It is a hybrid quantum-classical approach that minimizes an associated cost function in order to train a parameterized ansatz. In this paper we adapt the qualitative loss landscape characterization for neural networks introduced in \cite{goodfellow2014qualitatively,li2017visualizing} and tests for connectivity used in \cite{draxler2018essentially} to study the loss landscape features in PQC training. We present results for PQCs trained on…
The University of Tokyo（东京大学）；RIKEN（理化学研究所）；ENSAE（法国国家统计与经济管理学院）；CREST（法国国家经济统计与管理研究中心）
Recently, several studies have proven the global convergence and generalization abilities of the gradient descent method for two-layer ReLU networks. Most studies especially focused on the regression problems with the squared loss function, except for a few, and the importance of the positivity of the neural tangent kernel has been pointed out. On the other hand, the performance of gradient descent on classification problems using the logistic loss function has not been well studied, and further investigation of this problem structure is possible. In this work, we demonstrate that the separabi…
Precision and Intelligence Laboratory（精密智能实验室）；Tokyo Institute of Technology（东京工业大学）
In statistical problems, a set of parameterized probability distributions is used to estimate the true probability distribution. If Fisher information matrix at the true distribution is singular, then it has been left unknown what we can estimate about the true distribution from random samples. In this paper, we study a singular regression problem and prove a limit theorem which shows the relation between the singular regression problem and two birational invariants, a real log canonical threshold and a singular fluctuation. The obtained theorem has an important application to statistics, beca…
Policy gradient based reinforcement learning algorithms coupled with neural networks have shown success in learning complex policies in the model free continuous action space control setting. However, explicitly parameterized policies are limited by the scope of the chosen parametric probability distribution. We show that alternatively to the likelihood based policy gradient, a related objective can be optimized through advantage weighted quantile regression. Our approach models the policy implicitly in the network, which gives the agent the freedom to approximate any distribution in each acti…
CNRS（法国国家科学研究中心）；Aix-Marseille Université（艾克斯-马赛大学）；Université du Sud-Toulon-Var（南土伦瓦尔大学）
We present a novel approach to learn a kernel-based regression function. It is based on the useof conical combinations of data-based parameterized kernels and on a new stochastic convex optimization procedure of which we establish convergence guarantees. The overall learning procedure has the nice properties that a) the learned conical combination is automatically designed to perform the regression task at hand and b) the updates implicated by the optimization procedure are quite inexpensive. In order to shed light on the appositeness of our learning strategy, we present empirical results from…
In this paper, we propose a new semiparametric regression estimator by using a hybrid technique of a parametric approach and a nonparametric penalized spline method. The overall shape of the true regression function is captured by the parametric part, while its residual is consistently estimated by the nonparametric part. Asymptotic theory for the proposed semiparametric estimator is developed, showing that its behavior is dependent on the asymptotics for the nonparametric penalized spline estimator as well as on the discrepancy between the true regression function and the parametric part. As …
Wuhan University（武汉大学）；The Hong Kong University of Science and Technology（香港科技大学）
This paper studies the approximation capacity of ReLU neural networks with norm constraint on the weights. We prove upper and lower bounds on the approximation error of these networks for smooth function classes. The lower bound is derived through the Rademacher complexity of neural networks, which may be of independent interest. We apply these approximation bounds to analyze the convergences of regression using norm constrained neural networks and distribution estimation by GANs. In particular, we obtain convergence rates for over-parameterized neural networks. It is also shown that GANs can …
This paper rigorously establishes that the existence of the maximum likelihood estimate (MLE) in high-dimensional logistic regression models with Gaussian covariates undergoes a sharp `phase transition'. We introduce an explicit boundary curve $h_{\text{MLE}}$, parameterized by two scalars measuring the overall magnitude of the unknown sequence of regression coefficients, with the following property: in the limit of large sample sizes $n$ and number of features $p$ proportioned in such a way that $p/n \rightarrow κ$, we show that if the problem is sufficiently high dimensional in the sense tha…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Columbia University（哥伦比亚大学）；ETH Zürich（苏黎世联邦理工学院）
The generalization capacity of various machine learning models exhibits different phenomena in the under- and over-parameterized regimes. In this paper, we focus on regression models such as feature regression and kernel regression and analyze a generalized weighted least-squares optimization method for computational learning and inversion with noisy data. The highlight of the proposed framework is that we allow weighting in both the parameter space and the data space. The weighting scheme encodes both a priori knowledge on the object to be learned and a strategy to weight the contribution of …
University of Chicago（芝加哥大学）；Simons Institute, University of California at Berkeley（加州大学伯克利分校西蒙斯研究所）；University of British Columbia（不列颠哥伦比亚大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
We study a localized notion of uniform convergence known as an "optimistic rate" (Panchenko 2002; Srebro et al. 2010) for linear regression with Gaussian data. Our refined analysis avoids the hidden constant and logarithmic factor in existing results, which are known to be crucial in high-dimensional settings, especially for understanding interpolation learning. As a special case, our analysis recovers the guarantee from Koehler et al. (2021), which tightly characterizes the population risk of low-norm interpolators under the benign overfitting conditions. Our optimistic rate bound, though, al…
For massive data stored at multiple machines, we propose a distributed subsampling procedure for the composite quantile regression. By establishing the consistency and asymptotic normality of the composite quantile regression estimator from a general subsampling algorithm, we derive the optimal subsampling probabilities and the optimal allocation sizes under the L-optimality criteria. A two-step algorithm to approximate the optimal subsampling procedure is developed. The proposed methods are illustrated through numerical experiments on simulated and real datasets.
Locally optimal designs for generalized linear models are derived at certain values of the regression parameters. In the present paper a general setup of the generalized linear model is considered. Analytic solutions for optimal designs are developed under Kiefer $Φ_k$-criteria highlighting the D- and A-optimal designs. By means of The General Equivalence Theorem necessary and sufficient conditions in term of intensity values are obtained to characterize the locally optimal designs. In this context, linear predictors are assumed constituting first order models with and without intercept on app…
The paper considers model selection in regression under the additional structural constraints on admissible models where the number of potential predictors might be even larger than the available sample size. We develop a Bayesian formalism as a natural tool for generating a wide class of model selection criteria based on penalized least squares estimation with various complexity penalties associated with a prior on a model size. The resulting criteria are adaptive to structural constraints. We establish the upper bound for the quadratic risk of the resulting MAP estimator and the correspondin…
Regression models based on the log-symmetric family of distributions are particularly useful when the response is strictly positive and asymmetric. In this paper, we propose a class of quantile regression models based on reparameterized log-symmetric distributions, which have a quantile parameter. Two Monte Carlo simulation studies are carried out using the R software. The first one analyzes the performance of the maximum likelihood estimators, the information criteria AIC, BIC and AICc, and the generalized Cox-Snell and random quantile residuals. The second one evaluates the performance of th…
In this paper we propose a solution to the problem of parameter estimation of nonlinearly parameterized regressions--continuous or discrete time--and apply it for system identification and adaptive control. We restrict our attention to parameterizations that can be factorized as the product of two functions, a measurable one and a nonlinear function of the parameters to be estimated. Although in this case it is possible to define an extended vector of unknown parameters to get a linear regression, it is well-known that overparameterization suffers from some severe shortcomings. Another feature…
In this paper we propose a new methodology to represent the results of the robust ordinal regression approach by means of a family of representative value functions for which, taken two alternatives $a$ and $b$, the following two conditions are satisfied: 1) if for all compatible value functions $a$ is evaluated not worse than $b$ and for at least one value function $a$ has a better evaluation, then the evaluation of $a$ is greater than the evaluation of $b$ for all representative value functions; 2) if there exists one compatible value function giving $a$ an evaluation greater than $b$ and an…
This paper studies the identification of nonlinearly parameterized control systems in given experiments. Several identifiability criteria are established and an implementable algorithm is proposed for practicality with the convergence rate explicitly computed.
