We study the problem of linear regression where both covariates and responses are potentially (i) heavy-tailed and (ii) adversarially contaminated. Several computationally efficient estimators have been proposed for the simpler setting where the covariates are sub-Gaussian and uncontaminated; however, these estimators may fail when the covariates are either heavy-tailed or contain outliers. In this work, we show how to modify the Huber regression, least trimmed squares, and least absolute deviation estimators to obtain estimators which are simultaneously computationally and statistically efficient in the stronger contamination model. Our approach is quite simple, and consists of applying a filtering algorithm to the covariates, and then applying the classical robust regression estimators to the remaining data. We show that the Huber regression estimator achieves near-optimal error rates in this setting, whereas the least trimmed squares and least absolute deviation estimators can be made to achieve near-optimal error after applying a postprocessing step.
Robust Regression with Adaptive Contamination in Response: Optimal Rates and Computational Barriers
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Chicago（芝加哥大学）；University of California, San Diego（加州大学圣地亚哥分校）；Carnegie Mellon University（卡内基梅隆大学）
Robust methods have been a successful approach to deal with contaminations and noises in image processing. In this paper, we introduce a new robust method for two-dimensional autoregressive models. Our method, called BMM-2D, relies on representing a two-dimensional autoregressive process with an auxiliary model to attenuate the effect of contamination (outliers). We compare the performance of our method with existing robust estimators and the least squares estimator via a comprehensive Monte Carlo simulation study which considers different levels of replacement contamination and window sizes. …
A robust estimator for a wide family of mixtures of linear regression is presented. Robustness is based on the joint adoption of the Cluster Weighted Model and of an estimator based on trimming and restrictions. The selected model provides the conditional distribution of the response for each group, as in mixtures of regression, and further supplies local distributions for the explanatory variables. A novel version of the restrictions has been devised, under this model, for separately controlling the two sources of variability identified in it. This proposal avoids singularities in the log-lik…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
We study the problem of outlier robust high-dimensional mean estimation under a finite covariance assumption, and more broadly under finite low-degree moment assumptions. We consider a standard stability condition from the recent robust statistics literature and prove that, except with exponentially small failure probability, there exists a large fraction of the inliers satisfying this condition. As a corollary, it follows that a number of recently developed algorithms for robust mean estimation, including iterative filtering and non-convex gradient descent, give optimal error estimators with …
Cellwise outliers are likely to occur together with casewise outliers in modern data sets with relatively large dimension. Recent work has shown that traditional robust regression methods may fail for data sets in this paradigm. The proposed method, called three-step regression, proceeds as follows: first, it uses a consistent univariate filter to detect and eliminate extreme cellwise outliers; second, it applies a robust estimator of multivariate location and scatter to the filtered data to down-weight casewise outliers; third, it computes robust regression coefficients from the estimates obt…
The Gaussian cluster-weighted model (CWM) is a mixture of regression models with random covariates that allows for flexible clustering of a random vector composed of response variables and covariates. In each mixture component, it adopts a Gaussian distribution for both the covariates and the responses given the covariates. To robustify the approach with respect to possible elliptical heavy tailed departures from normality, due to the presence of atypical observations, the contaminated Gaussian CWM is here introduced. In addition to the parameters of the Gaussian CWM, each mixture component of…
Consider a linear model $Y=Xβ+z$, where $X=X_{n,p}$ and $z\sim N(0,I_n)$. The vector $β$ is unknown but is sparse in the sense that most of its coordinates are $0$. The main interest is to separate its nonzero coordinates from the zero ones (i.e., variable selection). Motivated by examples in long-memory time series (Fan and Yao [Nonlinear Time Series: Nonparametric and Parametric Methods (2003) Springer]) and the change-point problem (Bhattacharya [In Change-Point Problems (South Hadley, MA, 1992) (1994) 28-56 IMS]), we are primarily interested in the case where the Gram matrix $G=X'X$ is non…
In this paper a robust version of the classical Wald test statistics for linear hypothesis in the logistic regression model is introduced and its properties are explored. We study the problem under the assumption of random covariates although some ideas with non random covariates are also considered. The family of tests considered is based on the minimum density power divergence estimator instead of the maximum likelihood estimator and it is referred to as the Wald-type test statistic in the paper. We obtain the asymptotic distribution and also study the robustness properties of the Wald type …
Contaminated Multi-task Learning with Heterogeneity: Fundamental Limits and Optimal Algorithms
Department of Statistics, Pennsylvania State University（宾夕法尼亚州立大学统计学系）；School of Mathematics, University of Birmingham（伯明翰大学数学学院）；Department of Statistics, Columbia University（哥伦比亚大学统计学系）
Asymmetry along with heteroscedasticity or contamination often occurs with the growth of data dimensionality. In ultra-high dimensional data analysis, such irregular settings are usually overlooked for both theoretical and computational convenience. In this paper, we establish a framework for estimation in high-dimensional regression models using Penalized Robust Approximated quadratic M-estimators (PRAM). This framework allows general settings such as random errors lack of symmetry and homogeneity, or the covariates are not sub-Gaussian. To reduce the possible bias caused by the data's irregu…
The ordinary Bayes estimator based on the posterior density suffers from the potential problems of non-robustness under data contamination or outliers. In this paper, we consider the general set-up of independent but non-homogeneous (INH) observations and study a robustified pseudo-posterior based estimation for such parametric INH models. In particular, we focus on the $R^{(α)}$-posterior developed by Ghosh and Basu (2016) for IID data and later extended by Ghosh and Basu (2017) for INH set-up, where its usefulness and desirable properties have been numerically illustrated. In this paper, we …
