A current strand of research in high-dimensional statistics deals with robustifying the available methodology with respect to deviations from the pervasive light-tail assumptions. In this paper we consider a linear mean regression model with random design and potentially heteroscedastic, heavy-tailed errors, and investigate support estimation in this framework. We use a strictly convex, smooth variant of the Huber loss function with tuning parameter depending on the parameters of the problem, as well as the adaptive LASSO penalty for computational efficiency. For the resulting estimator we show sign-consistency and optimal rates of convergence in the $\ell_\infty$ norm as in the homoscedastic, light-tailed setting. In our analysis, we have to deal with the issue that the support of the target parameter in the linear mean regression model and its robustified version may differ substantially even for small values of the tuning parameter of the Huber loss function. Simulations illustrate the favorable numerical performance of the proposed methodology.
Despite its prevalence in statistical datasets, heteroscedasticity (non-constant sample variances) has been largely ignored in the high-dimensional statistics literature. Recently, studies have shown that the Lasso can accommodate heteroscedastic errors, with minor algorithmic modifications (Belloni et al., 2012; Gautier and Tsybakov, 2013). In this work, we study heteroscedastic regression with linear mean model and log-linear variances model with sparse high-dimensional parameters. In this work, we propose estimating variances in a post-Lasso fashion, which is followed by weighted-least squa…
The mixture of Dirichlet process (MDP) defines a flexible prior distribution on the space of probability measures. This study shows that ordinary least-squares (OLS) estimator, as a functional of the MDP posterior distribution, has posterior mean given by weighted least-squares (WLS), and has posterior covariance matrix given by the (weighted) heteroscedastic-consistent sandwich estimator. This is according to a pairs bootstrap distribution approximation of the posterior, using a Pólya urn scheme. Also, when the MDP prior baseline distribution is specified as a product of independent probabili…
In this article, we propose new Bayesian methods for selecting and estimating a sparse coefficient vector for skewed heteroscedastic response. Our novel Bayesian procedures effectively estimate the median and other quantile functions, accommodate non-local prior for regression effects without compromising ease of implementation via sampling based tools, and asymptotically select the true set of predictors even when the number of covariates increases in the same order of the sample size. We also extend our method to deal with some observations with very large errors. Via simulation studies and …
This paper studies the related problems of prediction, covariance estimation, and principal component analysis for the spiked covariance model with heteroscedastic noise. We consider an estimator of the principal components based on whitening the noise, and we derive optimal singular value and eigenvalue shrinkers for use with these estimated principal components. Underlying these methods are new asymptotic results for the high-dimensional spiked model with heteroscedastic noise, and consistent estimators for the relevant population parameters. We extend previous analysis on out-of-sample pred…
In this article we present an approach that enables joint wind speed and wind power forecasts for a wind park. We combine a multivariate seasonal time varying threshold autoregressive moving average (TVARMA) model with a power threshold generalized autoregressive conditional heteroscedastic (power-TGARCH) model. The modeling framework incorporates diurnal and annual periodicity modeling by periodic B-splines, conditional heteroscedasticity and a complex autoregressive structure with non-linear impacts. In contrast to usually time-consuming estimation approaches as likelihood estimation, we app…
Shrinkage algorithms are of great importance in almost every area of statistics due to the increasing impact of big data. Especially time series analysis benefits from efficient and rapid estimation techniques such as the lasso. However, currently lasso type estimators for autoregressive time series models still focus on models with homoscedastic residuals. Therefore, an iteratively reweighted adaptive lasso algorithm for the estimation of time series models under conditional heteroscedasticity is presented in a high-dimensional setting. The asymptotic behaviour of the resulting estimator is a…
Popular sparse estimation methods based on $\ell_1$-relaxation, such as the Lasso and the Dantzig selector, require the knowledge of the variance of the noise in order to properly tune the regularization parameter. This constitutes a major obstacle in applying these methods in several frameworks---such as time series, random fields, inverse problems---for which the noise is rarely homoscedastic and its level is hard to know in advance. In this paper, we propose a new approach to the joint estimation of the conditional mean and the conditional variance in a high-dimensional (auto-) regression s…
An adaptive nonparametric estimation procedure is constructed for heteroscedastic regression when the noise variance depends on the unknown regression. A non-asymptotic upper bound for a quadratic risk (oracle inequality) is obtained
An adaptive nonparametric estimation procedure is constructed for the estimation problem of heteroscedastic regression when the noise variance depends on the unknown regression. A non-asymptotic upper bound for a quadratic risk (an oracle inequality) is constructed.
Datasets with extreme observations and/or heavy-tailed error distributions are commonly encountered and should be analyzed with careful consideration of these features from a statistical perspective. Small deviations from an assumed model, such as the presence of outliers, can cause classical regression procedures to break down, potentially leading to unreliable inferences. Other distributional deviations, such as heteroscedasticity, can be handled by going beyond the mean and modelling the scale parameter in terms of covariates. We propose a method that accounts for heavy tails and heterosced…
This paper concerns the estimation of the regression function at a given point in nonparametric heteroscedastic models with Gaussian noise or with noise having unknown distribution. In the two cases an asymptotically efficient kernel estimator is constructed for the minimax absolute error risk.
We prove a new and general concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise. No specific structure is required on the model, except the existence of a suitable function that controls the local suprema of the empirical process. So far, only the case of linear contrast estimation was tackled in the literature with this level of generality on the model. We solve here the case of a quadratic contrast, by separating the behavior of a linearized empirical process and the empirical process driven by the squares of functions of model…
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper (2007) for estimation of unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk. It means that the asymptotic quadratic risk for this procedure coincides with a sharp lower bound.
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper, 2007, for estimating a unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk, i.e. the asymptotic quadratic risk for this procedure coincides with the Pinsker constant which gives a sharp lower bound for the quadratic risk over all possible estimators.
We consider the problem of choosing between several models in least-squares regression with heteroscedastic data. We prove that any penalization procedure is suboptimal when the penalty is a function of the dimension of the model, at least for some typical heteroscedastic model selection problems. In particular, Mallows' Cp is suboptimal in this framework. On the contrary, optimal model selection is possible with data-driven penalties such as resampling or $V$-fold penalties. Therefore, it is worth estimating the shape of the penalty from data, even at the price of a higher computational cost.…
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper, 2007, for estimating an unknown nonparametric regression. %\cite{GaPe1}. We prove that this procedure is asymptotically efficient for a quadratic risk, i.e. the asymptotic quadratic risk for this procedure coincides with the Pinsker constant which gives a sharp lower bound for the quadratic risk over all possible estimates
We consider the estimation of a regression function with random design and heteroscedastic noise in a nonparametric setting. More precisely, we address the problem of characterizing the optimal penalty when the regression function is estimated by using a penalized least-squares model selection method. In this context, we show the existence of a minimal penalty, defined to be the maximum level of penalization under which the model selection procedure totally misbehaves. The optimal penalty is shown to be twice the minimal one and to satisfy a non-asymptotic pathwise oracle inequality with leadi…
Heteroscedasticity is common in real world applications and is often handled by incorporating case weights into a modeling procedure. Intuitively, models fitted with different weight schemes would have a different level of complexity depending on how well the weights match the inverse of error variances. However, existing statistical theories on model complexity, also known as model degrees of freedom, were primarily established under the assumption of equal error variances. In this work, we focus on linear regression procedures and seek to extend the existing measures to a heteroscedastic set…
This paper provides general expression for Bartlett and Bartlett-type correction factors for the likelihood ratio and gradient statistics to test the dispersion parameter in heteroscedastic symmetric nonlinear models. This class of regression models is potentially useful for modeling data containing outlying observations. We consider a partition on the dispersion parameter vector in order to test the parameters of interest. Furthermore, we develop Monte Carlo simulations to compare the finite sample performances of the corrected tests proposed with the usual and modified score tests, likelihoo…
We consider the estimation of a bounded regression function with nonparametric heteroscedastic noise and random design. We study the true and empirical excess risks of the least-squares estimator on finite-dimensional vector spaces. We give upper and lower bounds on these quantities that are nonasymptotic and optimal to first order, allowing the dimension to depend on sample size. These bounds show the equivalence between the true and empirical excess risks when, among other things, the least-squares estimator is consistent in sup-norm with the projection of the regression function onto the co…
We consider a wavelet thresholding approach to adaptive variance function estimation in heteroscedastic nonparametric regression. A data-driven estimator is constructed by applying wavelet thresholding to the squared first-order differences of the observations. We show that the variance function estimator is nearly optimally adaptive to the smoothness of both the mean and variance functions. The estimator is shown to achieve the optimal adaptive rate of convergence under the pointwise squared error simultaneously over a range of smoothness classes. The estimator is also adaptively within a log…
This paper discusses asymptotic distributions of various estimators of the underlying parameters in some regression models with long memory (LM) Gaussian design and nonparametric heteroscedastic LM moving average errors. In the simple linear regression model, the first-order asymptotic distribution of the least square estimator of the slope parameter is observed to be degenerate. However, in the second order, this estimator is $n^{1/2}$-consistent and asymptotically normal for $h+H<3/2$; nonnormal otherwise, where $h$ and $H$ are LM parameters of design and error processes, respectively. The f…
We tackle the problem of building a prediction interval in heteroscedastic Gaussian regression. We focus on prediction intervals with constrained expected length in order to guarantee interpretability of the output. In this framework, we derive a closed form expression of the optimal prediction interval that allows for the development a data-driven prediction interval based on plug-in. The construction of the proposed algorithm is based on two samples, one labeled and another unlabeled. Under mild conditions, we show that our procedure is asymptotically as good as the optimal prediction interv…
We investigate function estimation in nonparametric regression models with random design and heteroscedastic correlated noise. Adaptive properties of warped wavelet nonlinear approximations are studied over a wide range of Besov scales, $f\in\mathcal{B}^s_{π,r}$, and for a variety of $L^p$ error measures. We consider error distributions with Long-Range-Dependence parameter $α,0<α\leq1$; heteroscedasticity is modeled with a design dependent function $σ$. We prescribe a tuning paradigm, under which warped wavelet estimation achieves partial or full adaptivity results with the rates that are show…
The paper concerns the problem of pointwise adaptive estimation in regression when the noise is heteroscedastic and incorrectly known. The use of the local approximation method, which includes the local polynomial smoothing as a particular case, leads to a finite family of estimators corresponding to different degrees of smoothing. Data-driven choice of localization degree in this case can be understood as the problem of selection from this family. This task can be performed by a suggested in Katkovnik and Spokoiny (2008) FLL technique based on Lepski's method. An important issue with this typ…
If uncorrelated random variables have a common expected value and decreasing variances then the variance of a sample mean is decreasing with the number of observations. Unfortunately, this natural and desirable Variance Reduction Property (VRP) by augmenting data is not automatically inherited by Ordinary Least Squares (OLS) estimators of parameters. In the paper we find conditions for the OLS to have the VRP. In the case of a straight line regression we show that the OLS estimators of intercept and slope have the VRP if the design points are increasing. This also holds true for alternating tw…
Parameter estimation in a class of heteroscedastic time series models is investigated. The existence of conditional least-squares and conditional likelihood estimators is proved. Their consistency and their asymptotic normality are established. Kernel estimators of the noise's density and its derivatives are defined and shown to be uniformly consistent. A simulation experiment conducted shows that the estimators perform well for large sample size.
We consider the common nonlinear regression model where the variance, as well as the mean, is a parametric function of the explanatory variables. The $c$-optimal design problem is investigated in the case when the parameters of both the mean and the variance function are of interest. A geometric characterization of $c$-optimal designs in this context is presented, which generalizes the classical result of Elfving [Ann. Math. Statist. 23 (1952) 255--262] for $c$-optimal designs. As in Elfving's famous characterization, $c$-optimal designs can be described as representations of boundary points o…
We consider the problem of designing experiments to detect the presence of a specified heteroscedastity in a non-linear Gaussian regression model. In this framework, we focus on the ${\rm D}_s$- and KL-criteria and study their relationship with the noncentrality parameter of the asymptotic chi-squared distribution of a likelihood-based test, for local alternatives. Specifically, we found that when the variance function depends just on one parameter, the two criteria coincide asymptotically and in particular, the ${\rm D}_1$-criterion is proportional to the noncentrality parameter. Differently,…
We investigate the optimality for model selection of the so-called slope heuristics, $V$-fold cross-validation and $V$-fold penalization in a heteroscedastic with random design regression context. We consider a new class of linear models that we call strongly localized bases and that generalize histograms, piecewise polynomials and compactly supported wavelets. We derive sharp oracle inequalities that prove the asymptotic optimality of the slope heuristics---when the optimal penalty shape is known---and $V$ -fold penalization. Furthermore, $V$-fold cross-validation seems to be suboptimal for a…
We prove the strong consistency and the asymptotic normality of the maximum likelihood estimator of the parameters of a general conditionally heteroscedastic model with $α$-stable innovations. Then, we relax the assumptions and only suppose that the innovation process converges in distribution toward a stable process. Using a pseudo maximum likelihood estimator with a stable density, we also obtain the strong consistency and the asymptotic normality of the estimator. This framework seems relevant for financial data exhibiting heavy tails. We apply this method to several financial index and com…
In many real applications, the distribution of measurement error could vary with each subject or even with each observation so the errors are heteroscedastic. In this paper, we propose a fast algorithm using a simulation-extrapolation (SIMEX) method to recover the unknown density in the case of heteroscedastic contamination. We show the consistency of the estimator and obtain its asymptotic variance and then address the practical selection of the smoothing parameter. We demonstrate that, through a finite sample simulation study, the proposed method performs better than the Fourier-type deconvo…
Completely nonparametric transformation models with heteroscedastic errors are considered. Despite their flexibility, such models have rarely been used so far, since estimators of the model components have been missing and even identification of such models has not been clear until very recently. The results of Kloodt (2020) are used to construct the first two estimators of the transformation function in these models. While the first estimator converges to the true transformation function at a parametric rate, the second estimator can be obtained by an explicit formula and is less computationa…
Carnegie Mellon University（卡内基梅隆大学）；University of Florida（佛罗里达大学）
We consider least squares estimation in a general nonparametric regression model. The rate of convergence of the least squares estimator (LSE) for the unknown regression function is well studied when the errors are sub-Gaussian. We find upper bounds on the rates of convergence of the LSE when the errors have uniformly bounded conditional variance and have only finitely many moments. We show that the interplay between the moment assumptions on the error, the metric entropy of the class of functions involved, and the "local" structure of the function class around the truth drives the rate of con…
We introduce a robust and fully adaptive method for pointwise estimation in heteroscedastic regression. We allow for noise and design distributions that are unknown and fulfill very weak assumptions only. In particular, we do not impose moment conditions on the noise distribution. Moreover, we do not require a positive density for the design distribution. In a first step, we study the consistency of locally polynomial M-estimators that consist of a contrast and a kernel. Afterwards, minimax results are established over unidimensional Hölder spaces for degenerate design. We then choose the cont…
We revisit the model of heteroscedastic extremes initially introduced by Einmahl et al. (JRSSB, 2016) to describe the evolution of a non stationary sequence whose extremes evolve over time and adapt it into a general extreme quantile regression framework. We provide estimates for the extreme value index and the integrated skedasis function and prove their asymptotic normality. Our results are quite similar to those developed for heteroscedastic extremes but with a different proof approach emphasizing coupling arguments. We also propose a pointwise estimator of the skedasis function and a Weiss…
In this paper we consider a heteroscedastic transformation model, where the transformation belongs to a parametric family of monotone transformations, the regression and variance function are modelled nonparametrically and the error is independent of the multidimensional covariates. In this model, we first consider the estimation of the unknown components of the model, namely the transformation parameter, regression and variance function and the distribution of the error. We show the asymptotic normality of the proposed estimators. Second, we propose tests for the validity of the model, and es…
It is common, in deconvolution problems, to assume that the measurement errors are identically distributed. In many real-life applications, however, this condition is not satisfied and the deconvolution estimators developed for homoscedastic errors become inconsistent. In this paper, we introduce a kernel estimator of a density in the case of heteroscedastic contamination. We establish consistency of the estimator and show that it achieves optimal rates of convergence under quite general conditions. We study the limits of application of the procedure in some extreme situations, where we show t…
This paper studies the quasi-maximum-likelihood estimator (QMLE) in a general conditionally heteroscedastic time series model of multiplicative form $X_t=σ_tZ_t$, where the unobservable volatility $σ_t$ is a parametric function of $(X_{t-1},...,X_{t-p},σ_{t-1},... ,σ_{t-q})$ for some $p,q\ge0$, and $(Z_t)$ is standardized i.i.d. noise. We assume that these models are solutions to stochastic recurrence equations which satisfy a contraction (random Lipschitz coefficient) property. These assumptions are satisfied for the popular GARCH, asymmetric GARCH and exponential GARCH processes. Exploiting …
We observe $n$ heteroscedastic stochastic processes $\{Y_v(t)\}_{v}$, where for any $v\in\{1,\ldots,n\}$ and $t \in [0,1]$, $Y_v(t)$ is the convolution product of an unknown function $f$ and a known blurring function $g_v$ corrupted by Gaussian noise. Under an ordinary smoothness assumption on $g_1,\ldots,g_n$, our goal is to estimate the $d$-th derivatives (in weak sense) of $f$ from the observations. We propose an adaptive estimator based on wavelet block thresholding, namely the "BlockJS estimator". Taking the mean integrated squared error (MISE), our main theoretical result investigates th…
In this paper, a new family of resampling-based penalization procedures for model selection is defined in a general framework. It generalizes several methods, including Efron's bootstrap penalization and the leave-one-out penalization recently proposed by Arlot (2008), to any exchangeable weighted bootstrap resampling scheme. In the heteroscedastic regression framework, assuming the models to have a particular structure, these resampling penalties are proved to satisfy a non-asymptotic oracle inequality with leading constant close to 1. In particular, they are asympotically optimal. Resampling…
Let $Y$ be a Gaussian vector of $\mathbb{R}^n$ of mean $s$ and diagonal covariance matrix $Γ$. Our aim is to estimate both $s$ and the entries $σ_i=Γ_{i,i}$, for $i=1,...,n$, on the basis of the observation of two independent copies of $Y$. Our approach is free of any prior assumption on $s$ but requires that we know some upper bound $γ$ on the ratio $\max_iσ_i/\min_iσ_i$. For example, the choice $γ=1$ corresponds to the homoscedastic case where the components of $Y$ are assumed to have common (unknown) variance. In the opposite, the choice $γ>1$ corresponds to the heteroscedastic case where t…
McGill University（麦吉尔大学）；Google Research（谷歌研究院）；Pompeu Fabra University（庞培法布拉大学）；ICREA（加泰罗尼亚高等研究院）；Barcelona Graduate School of Economics（巴塞罗那经济学研究生院）
We study the problem of estimating the common mean $μ$ of $n$ independent symmetric random variables with different and unknown standard deviations $σ_1 \le σ_2 \le \cdots \leσ_n$. We show that, under some mild regularity assumptions on the distribution, there is a fully adaptive estimator $\widehatμ$ such that it is invariant to permutations of the elements of the sample and satisfies that, up to logarithmic factors, with high probability, \[ |\widehatμ - μ| \lesssim \min\left\{σ_{m^*}, \frac{\sqrt{n}}{\sum_{i = \sqrt{n}}^n σ_i^{-1}} \right\}~, \] where the index $m^* \lesssim \sqrt{n}$ satis…
This paper is to prove the asymptotic normality of a statistic for detecting the existence of heteroscedasticity for linear regression models without assuming randomness of covariates when the sample size $n$ tends to infinity and the number of covariates $p$ is either fixed or tends to infinity. Moreover our approach indicates that its asymptotic normality holds even without homoscedasticity.
This paper aims to address two fundamental challenges arising in eigenvector estimation and inference for a low-rank matrix from noisy observations: (1) how to estimate an unknown eigenvector when the eigen-gap (i.e. the spacing between the associated eigenvalue and the rest of the spectrum) is particularly small; (2) how to perform estimation and inference on linear functionals of an eigenvector -- a sort of "fine-grained" statistical reasoning that goes far beyond the usual $\ell_2$ analysis. We investigate how to address these challenges in a setting where the unknown $n\times n$ matrix is …
Consider the problem of estimating a multivariate normal mean with a known variance matrix, which is not necessarily proportional to the identity matrix. The coordinates are shrunk directly in proportion to their variances in Efron and Morris' (J. Amer. Statist. Assoc. 68 (1973) 117-130) empirical Bayes approach, whereas inversely in proportion to their variances in Berger's (Ann. Statist. 4 (1976) 223-226) minimax estimators. We propose a new minimax estimator, by approximately minimizing the Bayes risk with a normal prior among a class of minimax estimators where the shrinkage direction is o…
Estimation for the log-logistic and Weibull distributions can be performed by using the equations used for probability plotting. The equations leads to highly heteroscedastic regression. Exact expressions for the variances of the residuals are derived which can be used to perform weighted regression. In large samples maximum likelihood performs best, but it is shown that in smaller samples the weighted regression outperforms maximum likelihood estimation with respect to bias and mean square error.
