Neural networks are becoming an increasingly important tool in applications. However, neural networks are not widely used in statistical genetics. In this paper, we propose a new neural networks method called expectile neural networks. When the size of parameter is too large, the standard maximum likelihood procedures may not work. We use sieve method to constrain parameter space. And we prove its consistency and normality under nonparametric regression framework.
Expectiles were introduced by Newey and Powell (1987) in the context of linear regression models. Recently, Bellini et al. (2014) revealed that expectiles can also be seen as reasonable law-invariant risk measures. In this article, we show that the corresponding statistical functionals are continuous w.r.t. the $1$-weak topology and suitably functionally differentiable. By means of these regularity results we can derive several properties as consistency, asymptotic normality, bootstrap consistency, and qualitative robustness of the corresponding estimators in nonparametric and parametric stati…
We discuss in detail the asymptotic distribution of sample expectiles. First, we show uniform consistency under the assumption of a finite mean. In case of a finite second moment, we show that for expectiles other then the mean, only the additional assumption of continuity of the distribution function at the expectile implies asymptotic normality, otherwise, the limit is non-normal. For a continuous distribution function we show the uniform central limit theorem for the expectile process. If, in contrast, the distribution is heavy-tailed, and contained in the domain of attraction of a stable l…
In the literature, quite a few measures have been proposed for quantifying the deviation of a probability distribution from symmetry. The most popular of these skewness measures are based on the third centralized moment and on quantiles. However, there are major drawbacks in using these quantities. These include a strong emphasis on the distributional tails and a poor asymptotic behaviour for the (empirical) moment based measure as well as difficult statistical inference and strange behaviour for discrete distributions for quantile based measures. Therefore, in this paper, we introduce skewnes…
In the present paper we address the real-time detection problem of a change-point in the coefficients of a linear model with the possibility that the model errors are asymmetrical and that the explanatory variables number is large. We build test statistics based on the cumulative sum (CUSUM) of the expectile function derivatives calculated on the residuals obtained by the expectile and adaptive LASSO expectile estimation methods. The asymptotic distribution of these statistics are obtained under the hypothesis that the model does not change. Moreover, we prove that they diverge when the model …
The paper focuses on the automatic selection of the grouped explanatory variables in an high-dimensional model, when the model errors are asymmetric. After introducing the model and notations, we define the adaptive group LASSO expectile estimator for which we prove the oracle properties: the sparsity and the asymptotic normality. Afterwards, the results are generalized by considering the asymmetric $L_q$-norm loss function. The theoretical results are obtained in several cases with respect to the number of variable groups. This number can be fixed or dependent on the sample size $n$, with the…
The identification of influential observations is an important part of data analysis that can prevent erroneous conclusions drawn from biased estimators. However, in high dimensional data, this identification is challenging. Classical and recently-developed methods often perform poorly when there are multiple influential observations in the same dataset. In particular, current methods can fail when there is masking several influential observations with similar characteristics, or swamping when the influential observations are near the boundary of the space spanned by well-behaved observations.…
National Technical University of Athens（雅典国立技术大学）；Hellenic Air Force（希腊空军）；World Meteorological Organisation (WMO)（世界气象组织）；University of the West of England（西英格兰大学）
Neural networks are one of the most popularly used methods in machine learning and artificial intelligence nowadays. Due to the universal approximation theorem (Hornik et al. (1989)), a neural network with one hidden layer can approximate any continuous function on a compact support as long as the number of hidden units is sufficiently large. Statistically, a neural network can be classified into a nonlinear regression framework. However, if we consider it parametrically, due to the unidentifiability of the parameters, it is difficult to derive its asymptotic properties. Instead, we considered…
We study the properties of nonparametric least squares regression using deep neural networks. We derive non-asymptotic upper bounds for the prediction error of the empirical risk minimizer of feedforward deep neural regression. Our error bounds achieve minimax optimal rate and significantly improve over the existing ones in the sense that they depend polynomially on the dimension of the predictor, instead of exponentially on dimension. We show that the neural regression estimator can circumvent the curse of dimensionality under the assumption that the predictor is supported on an approximate l…
We study the asymptotic theory of misspecified models for diffusion processes with noisy nonsynchronous observations. Unlike with correctly specified models, the original maximum-likelihood-type estimator has an asymptotic bias under the misspecified setting and fails to achieve an optimal rate of convergence. To address this, we consider a new quasi-likelihood function that arrows constructing a maximum-likelihood-type estimator that achieves the optimal rate of convergence. Study of misspecified models enables us to apply machine-learning techniques to the maximum-likelihood approach. With t…
This paper discusses the asymptotic behavior of regression models under general conditions. First, we give a general inequality for the difference of the sum of square errors (SSE) of the estimated regression model and the SSE of the theoretical best regression function in our model. A set of generalized derivative functions is a key tool in deriving such inequality. Under suitable Donsker condition for this set, we give the asymptotic distribution for the difference of SSE. We show how to get this Donsker property for parametric models even if the parameters characterizing the best regression…
Statistical divergences (SDs), which quantify the dissimilarity between probability distributions, are a basic constituent of statistical inference and machine learning. A modern method for estimating those divergences relies on parametrizing an empirical variational form by a neural network (NN) and optimizing over parameter space. Such neural estimators are abundantly used in practice, but corresponding performance guarantees are partial and call for further exploration. We establish non-asymptotic absolute error bounds for a neural estimator realized by a shallow NN, focusing on four popula…
Neural networks (NN) play a central role in modern Artificial intelligence (AI) technology and has been successfully used in areas such as natural language processing and image recognition. While majority of NN applications focus on prediction and classification, there are increasing interests in studying statistical inference of neural networks. The study of NN statistical inference can enhance our understanding of NN statistical proprieties. Moreover, it can facilitate the NN-based hypothesis testing that can be applied to hypothesis-driven clinical and biomedical research. In this paper, we…
In this paper, we consider a model called CHARME (Conditional Heteroscedastic Autoregressive Mixture of Experts), a class of generalized mixture of nonlinear nonparametric AR-ARCH time series. Under certain Lipschitz-type conditions on the autoregressive and volatility functions, we prove that this model is stationary, ergodic and $τ$-weakly dependent. These conditions are much weaker than those presented in the literature that treats this model. Moreover, this result forms the theoretical basis for deriving an asymptotic theory of the underlying (non)parametric estimation, which we present fo…
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；Department of Chemistry, Stanford University（斯坦福大学化学系）；Center for Data Science, New York University（纽约大学数据科学中心）
Recent theoretical works have characterized the dynamics of wide shallow neural networks trained via gradient descent in an asymptotic mean-field limit when the width tends towards infinity. At initialization, the random sampling of the parameters leads to deviations from the mean-field limit dictated by the classical Central Limit Theorem (CLT). However, since gradient descent induces correlations among the parameters, it is of interest to analyze how these fluctuations evolve. Here, we use a dynamical CLT to prove that the asymptotic fluctuations around the mean limit remain bounded in mean …
The Hong Kong Polytechnic University（香港理工大学）；Wuhan University（武汉大学）；The Chinese University of Hong Kong（香港中文大学）
We propose a penalized nonparametric approach to estimating the quantile regression process (QRP) in a nonseparable model using rectifier quadratic unit (ReQU) activated deep neural networks and introduce a novel penalty function to enforce non-crossing of quantile regression curves. We establish the non-asymptotic excess risk bounds for the estimated QRP and derive the mean integrated squared error for the estimated QRP under mild smoothness and regularity conditions. To establish these non-asymptotic risk and estimation error bounds, we also develop a new error bound for approximating $C^s$ …
We present a unified mathematical derivation of the asymptotic behaviour of three of the main forms of \textit{directed transfer function} (DTF) complementing recent partial directed coherence (PDC) results \cite{Baccala2013}. Based on these results and numerical examples we argue for a new directed `link' centered neural connectivity framework to replace the widespread correlation based effective/functional network concepts so that directed network influences between structures become classified as to whether links are \textit{active} in a \textit{direct} or in an \textit{indirect} way thereb…
Statistical distances (SDs), which quantify the dissimilarity between probability distributions, are central to machine learning and statistics. A modern method for estimating such distances from data relies on parametrizing a variational form by a neural network (NN) and optimizing it. These estimators are abundantly used in practice, but corresponding performance guarantees are partial and call for further exploration. In particular, there seems to be a fundamental tradeoff between the two sources of error involved: approximation and estimation. While the former needs the NN class to be rich…
The Chinese University of Hong Kong（香港中文大学）；Wuhan University（武汉大学）；University of Iowa（爱荷华大学）
In this paper, we consider the problem of binary classification with a class of general deep convolutional neural networks, which includes fully-connected neural networks and fully convolutional neural networks as special cases. We establish non-asymptotic excess risk bounds for a class of convex surrogate losses and target functions with different modulus of continuity. An important feature of our results is that we clearly define the prefactors of the risk bounds in terms of the input data dimension and other model parameters and show that they depend polynomially on the dimensionality in so…
Neural networks have shown great predictive power when dealing with various unstructured data such as images and natural languages. The Bayesian neural network captures the uncertainty of prediction by putting a prior distribution for the parameter of the model and computing the posterior distribution. In this paper, we show that the Bayesian neural network using spike-and-slab prior has consistency with nearly minimax convergence rate when the true regression function is in the Besov space. Even when the smoothness of the regression function is unknown the same posterior convergence rate hold…
Rutgers University（罗格斯大学）；The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所人工智能项目）
We investigate the asymptotic risk of a general class of overparameterized likelihood models, including deep models. The recent empirical success of large-scale models has motivated several theoretical studies to investigate a scenario wherein both the number of samples, $n$, and parameters, $p$, diverge to infinity and derive an asymptotic risk at the limit. However, these theorems are only valid for linear-in-feature models, such as generalized linear regression, kernel regression, and shallow neural networks. Hence, it is difficult to investigate a wider class of nonlinear models, including…
The space time autoregressive model has been widely applied in science, in areas such as economics, public finance, political science, agricultural economics, environmental studies and transportation analyses. The classical space time autoregressive model is a linear model for describing spatial correlation. In this work, we expand the classical model to include related exogenous variables, possibly non-Gaussian, high volatility errors, and a nonlinear neural network component. The nonlinear neural network component allows for more model flexibility, the ability to learn and model nonlinear an…
University of Torino（都灵大学）；Collegio Carlo Alberto（卡洛·阿尔贝托学院）；Bocconi University（博科尼大学）；Cogent Labs（Cogent实验室）
There is a recent and growing literature on large-width asymptotic properties of Gaussian neural networks (NNs), namely NNs whose weights are initialized as Gaussian distributions. Two popular problems are: i) the study of the large-width distributions of NNs, which characterizes the infinitely wide limit of a rescaled NN in terms of a Gaussian stochastic process; ii) the study of the large-width training dynamics of NNs, which characterizes the infinitely wide dynamics in terms of a deterministic kernel, referred to as the neural tangent kernel (NTK), and shows that, for a sufficiently large …
University of Torino（都灵大学）；Collegio Carlo Alberto（卡洛·阿尔贝托学院）；Bocconi University（博科尼大学）；Cogent Labs（Cogent实验室）
In modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotic properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and Gaussian stochastic processes (SPs). Such an interplay has proved to be critical in Bayesian inference under Gaussian SP priors, kernel regression for infinitely wide deep NNs trained via gradient descent, and information propagation within infinitely wide NNs. Motivated by empirical analyses that show the potential of replacing Gaussian distributions with Stable distributions fo…
Yale University（耶鲁大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Department of Statistics and Data Science（统计与数据科学系）；Department of Mathematics（数学系）
We study the eigenvalue distributions of the Conjugate Kernel and Neural Tangent Kernel associated to multi-layer feedforward neural networks. In an asymptotic regime where network width is increasing linearly in sample size, under random initialization of the weights, and for input samples satisfying a notion of approximate pairwise orthogonality, we show that the eigenvalue distributions of the CK and NTK converge to deterministic limits. The limit for the CK is described by iterating the Marcenko-Pastur map across the hidden layers. The limit for the NTK is equivalent to that of a linear co…
Fast Gradient Sign Method (FGSM) is a popular method to generate adversarial examples that make neural network models robust against perturbations. Despite its empirical success, its theoretical property is not well understood. This paper develops theory to explain the regularization effect of Generalized FGSM, a class of methods to generate adversarial examples. Motivated from the relationship between FGSM and LASSO penalty, the asymptotic properties of Generalized FGSM are derived in the Generalized Linear Model setting, which is essentially the 1-layer neural network setting with certain ac…
Asymptotics of Learning with Deep Structured (Random) Features
ETH Zurich（苏黎世联邦理工学院）；EPFL（洛桑联邦理工学院）；École Normale Supérieure (ENS) - PSL（巴黎高等师范学院（PSL大学））；CNRS（法国国家科学研究中心）；ETH AI Center（苏黎世联邦理工学院人工智能中心）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
