Functional linear regression is a widely used approach to model functional responses with respect to functional inputs. However, classical functional linear regression models can be severely affected by outliers. We therefore introduce a Fisher-consistent robust functional linear regression model that is able to effectively fit data in the presence of outliers. The model is built using robust functional principal component and least squares regression estimators. The performance of the functional linear regression model depends on the number of principal components used. We therefore introduce a consistent robust model selection procedure to choose the number of principal components. Our robust functional linear regression model can be used alongside an outlier detection procedure to effectively identify abnormal functional responses. A simulation study shows our method is able to effectively capture the regression behaviour in the presence of outliers, and is able to find the outliers with high accuracy. We demonstrate the usefulness of our method on jet engine sensor data. We identify outliers that would not be found if the functional responses were modelled independently of the functional input, or using non-robust methods.
Many biological high-throughput data sets, such as targeted amplicon-based and metagenomic sequencing data, are compositional in nature. A common exploratory data analysis task is to infer statistical associations between the high-dimensional microbial compositions and habitat- or host-related covariates. We propose a general robust statistical regression framework, RobRegCC (Robust Regression with Compositional Covariates), which extends the linear log-contrast model by a mean shift formulation for capturing outliers. RobRegCC includes sparsity-promoting convex and non-convex penalties for pa…
We develop a new robust geographically weighted regression method in the presence of outliers. We embed the standard geographically weighted regression in robust objective function based on $γ$-divergence. A novel feature of the proposed approach is that two tuning parameters that control robustness and spatial smoothness are automatically tuned in a data-dependent manner. Further, the proposed method can produce robust standard error estimates of the robust estimator and give us a reasonable quantity for local outlier detection. We demonstrate that the proposed method is superior to the exist…
Robust estimation and variable selection procedure are developed for the extended t-process regression model with functional data. Statistical properties such as consistency of estimators and predictions are obtained. Numerical studies show that the proposed method performs well.
A robust estimator is proposed for the parameters that characterize the linear regression problem. It is based on the notion of shrinkages, often used in Finance and previously studied for outlier detection in multivariate data. A thorough simulation study is conducted to investigate: the efficiency with normal and heavy-tailed errors, the robustness under contamination, the computational times, the affine equivariance and breakdown value of the regression estimator. Two classical data-sets often used in the literature and a real socio-economic data-set about the Living Environment Deprivation…
To conduct regression analysis for data contaminated with outliers, many approaches have been proposed for simultaneous outlier detection and robust regression, so is the approach proposed in this manuscript. This new approach is called "penalized weighted least squares" (PWLS). By assigning each observation an individual weight and incorporating a lasso-type penalty on the log-transformation of the weight vector, the PWLS is able to perform outlier detection and robust regression simultaneously. A Bayesian point-of-view of the PWLS is provided, and it is showed that the PWLS can be seen as an…
The errors-in-variables (EIV) regression model, being more realistic by accounting for measurement errors in both the dependent and the independent variables, is widely adopted in applied sciences. The traditional EIV model estimators, however, can be highly biased by outliers and other departures from the underlying assumptions. In this paper, we develop a novel nonparametric regression approach - the robust compound regression (RCR) analysis method for the robust estimation of EIV models. We first introduce a robust and efficient estimator called least sine squares (LSS). Taking full advanta…
A new type of redescending M-estimators is constructed, based on data augmentation with an unspecified outlier model. Necessary and sufficient conditions for the convergence of the resulting estimators to the Hubertype skipped mean are derived. By introducing a temperature parameter the concept of deterministic annealing can be applied, making the estimator insensitive to the starting point of the iteration. The properties of the annealing M-estimator as a function of the temperature are explored. Finally, two applications are presented. The first one is the robust estimation of interaction ve…
Beta regression models are widely used for modeling continuous data limited to the unit interval, such as proportions, fractions, and rates. The inference for the parameters of beta regression models is commonly based on maximum likelihood estimation. However, it is known to be sensitive to discrepant observations. In some cases, one atypical data point can lead to severe bias and erroneous conclusions about the features of interest. In this work, we develop a robust estimation procedure for beta regression models based on the maximization of a reparameterized Lq-likelihood. The new estimator …
We consider the nonparametric robust estimation problem for regression models in continuous time with semi-Markov noises. An adaptive model selection procedure is proposed. Under general moment conditions on the noise distribution a sharp non-asymptotic oracle inequality for the robust risks is obtained and the robust efficiency is shown. It turns out that for semi-Markov models the robust minimax convergence rate may be faster or slower than the classical one.
Cox proportional hazard regression model is a popular tool to analyze the relationship between a censored lifetime variable with other relevant factors. The semi-parametric Cox model is widely used to study different types of data arising from applied disciplines like medical science, biology, reliability studies and many more. A fully parametric version of the Cox regression model, if properly specified, can yield more efficient parameter estimates leading to better insight generation. However, the existing maximum likelihood approach of generating inference under the fully parametric Cox reg…
Institute for Mathematics, Carl von Ossietzky University Oldenburg（奥尔登堡卡尔·冯·奥西茨基大学数学研究所）
Neural networks are an indispensable model class for many complex learning tasks. Despite the popularity and importance of neural networks and many different established techniques from literature for stabilization and robustification of the training, the classical concepts from robust statistics have rarely been considered so far in the context of neural networks. Therefore, we adapt the notion of the regression breakdown point to regression neural networks and compute the breakdown point for different feed-forward network configurations and contamination settings. In an extensive simulation …
In this article we consider the nonparametric robust estimation problem for regression models in continuous time with semi-Markov noises observed in discrete time moments. An adaptive model selection procedure is proposed. A sharp non-asymptotic oracle inequality for the robust risks is obtained. We obtain sufficient conditions on the frequency observations under which the robust efficiency is shown. It turns out that for the semi-Markov models the robust minimax convergence rate may be faster or slower than the classical one.
We consider new formulations and methods for sparse quantile regression in the high-dimensional setting. Quantile regression plays an important role in many applications, including outlier-robust exploratory analysis in gene selection. In addition, the sparsity consideration in quantile regression enables the exploration of the entire conditional distribution of the response variable given the predictors and therefore yields a more comprehensive view of the important predictors. We propose a generalized OMP algorithm for variable selection, taking the misfit loss to be either the traditional q…
We present a Distributionally Robust Optimization (DRO) approach to estimate a robustified regression plane in a linear regression setting, when the observed samples are potentially contaminated with adversarially corrupted outliers. Our approach mitigates the impact of outliers through hedging against a family of distributions on the observed data, some of which assign very low probabilities to the outliers. The set of distributions under consideration are close to the empirical distribution in the sense of the Wasserstein metric. We show that this DRO formulation can be relaxed to a convex o…
Ordinary least square (OLS), maximum likelihood (ML) and robust methods are the widely used methods to estimate the parameters of a linear regression model. It is well known that these methods perform well under some distributional assumptions on error terms. However, these distributional assumptions on the errors may not be appropriate for some data sets. In these case, nonparametric methods may be considered to carry on the regression analysis. Empirical likelihood (EL) method is one of these nonparametric methods. The EL method maximizes a function, which is multiplication of the unknown pr…
Nagoya Institute of Technology（名古屋工业大学）；RIKEN（理化学研究所）
In practical data analysis under noisy environment, it is common to first use robust methods to identify outliers, and then to conduct further analysis after removing the outliers. In this paper, we consider statistical inference of the model estimated after outliers are removed, which can be interpreted as a selective inference (SI) problem. To use conditional SI framework, it is necessary to characterize the events of how the robust method identifies outliers. Unfortunately, the existing methods cannot be directly used here because they are applicable to the case where the selection events c…
We study fast algorithms for statistical regression problems under the strong contamination model, where the goal is to approximately optimize a generalized linear model (GLM) given adversarially corrupted samples. Prior works in this line of research were based on the robust gradient descent framework of Prasad et. al., a first-order method using biased gradient queries, or the Sever framework of Diakonikolas et. al., an iterative outlier-removal method calling a stationary point finder. We present nearly-linear time algorithms for robust regression problems with improved runtime or estimatio…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Cambridge（剑桥大学）
We study the problem of linear regression where both covariates and responses are potentially (i) heavy-tailed and (ii) adversarially contaminated. Several computationally efficient estimators have been proposed for the simpler setting where the covariates are sub-Gaussian and uncontaminated; however, these estimators may fail when the covariates are either heavy-tailed or contain outliers. In this work, we show how to modify the Huber regression, least trimmed squares, and least absolute deviation estimators to obtain estimators which are simultaneously computationally and statistically effic…
In this work we revisit two classic high-dimensional online learning problems, namely linear regression and contextual bandits, from the perspective of adversarial robustness. Existing works in algorithmic robust statistics make strong distributional assumptions that ensure that the input data is evenly spread out or comes from a nice generative model. Is it possible to achieve strong robustness guarantees even without distributional assumptions altogether, where the sequence of tasks we are asked to solve is adaptively and adversarially chosen? We answer this question in the affirmative for b…
We discuss an approach for fitting robust nonlinear regression models, which can be employed to model and predict the contagion dynamics of the Covid-19 in Italy. The focus is on the analysis of epidemic data using robust dose-response curves, but the functionality is applicable to arbitrary nonlinear regression models.
The scalar-on-function regression model has become a popular analysis tool to explore the relationship between a scalar response and multiple functional predictors. Most of the existing approaches to estimate this model are based on the least-squares estimator, which can be seriously affected by outliers in empirical datasets. When outliers are present in the data, it is known that the least-squares-based estimates may not be reliable. This paper proposes a robust functional partial least squares method, allowing a robust estimate of the regression coefficients in a scalar-on-multiple-function…
The bridge regression estimator generalizes both ridge regression and LASSO estimators. Since it minimizes the sum of squared residuals with a $L_{γ}$ penalty, this estimator is typically not robust against outliers in the data. There have been attempts to define robust versions of the bridge regression method, but while these proposed methods produce bridge regression estimators robust to outliers and heavy-tailed errors, they are not robust against leverage points. We propose a robust bridge regression estimation method combining MM and bridge regression estimation methods. The MM bridge reg…
The Seemingly Unrelated Regressions (SUR) model is a wide used estimation procedure in econometrics, insurance and finance, where very often, the regression model contains more than one equation. Unknown parameters, regression coefficients and covariances among the errors terms, are estimated using algorithms based on Generalized Least Squares or Maximum Likelihood, and the method, as a whole, is very sensitive to outliers. To overcome this problem M-estimators and S-estimators are proposed in the literature together with fast algorithms. However, these procedures are only able to cope with ro…
Seemingly unrelated regression models generalize linear regression models by considering multiple regression equations that are linked by contemporaneously correlated disturbances. Robust inference for seemingly unrelated regression models is considered. MM-estimators are introduced to obtain estimators that have both a high breakdown point and a high normal efficiency. A fast and robust bootstrap procedure is developed to obtain robust inference for these estimators. Confidence intervals for the model parameters as well as hypothesis tests for linear restrictions of the regression coefficient…
In other FICO Technical Papers, I have shown how to fit Generalized Additive Models (GAM) with shape constraints using quadratic programming applied to B-Spline component functions. In this paper, I extend the method to Robust Least Squares Regression.
We give methods for the construction of designs for linear models, when the purpose of the investigation is the estimation of the conditional quantile function and the estimation method is quantile regression. The designs are robust against misspecified response functions, and against unanticipated heteroscedasticity. The methods are illustrated by example, and in a case study in which they are applied to growth charts.
In this article, we introduce a new variable selection technique through trimming for finite mixture of regression models. Compared to the traditional variable selection techniques, the new method is robust and not sensitive to outliers. The estimation algorithm is introduced and numerical studies are conducted to examine the finite sample performance of the proposed procedure and to compare it with other existing methods.
We present a robust regression estimator for longitudinal data, which is especially suited for functional data that has been observed on sparse or irregular time grids. We show by simulation that the proposed estimators possess good outlier-resistance properties compared with the traditional functional least-squares estimator. As an example of application, we study the relationship between levels of oxides of nitrogen and ozone in the city of San Francisco.
Although linear regression models are fundamental tools in statistical science, the estimation results can be sensitive to outliers. While several robust methods have been proposed in frequentist frameworks, statistical inference is not necessarily straightforward. We here propose a Bayesian approach to robust inference on linear regression models using synthetic posterior distributions based on $γ$-divergence, which enables us to naturally assess the uncertainty of the estimation through the posterior distribution. We also consider the use of shrinkage priors for the regression coefficients t…
Gaussian process regression (GPR) model is well-known to be susceptible to outliers. Robust process regression models based on t-process or other heavy-tailed processes have been developed to address the problem. However, due to the nature of the current definition for heavy-tailed processes, the unknown process regression function and the random errors are always defined jointly and thus dependently. This definition, mainly owing to the dependence assumption involved, is not justified in many practical problems and thus limits the application of those robust approaches. It also results in a l…
The cellwise robust M regression estimator is introduced as the first estimator of its kind that intrinsically yields both a map of cellwise outliers consistent with the linear model, and a vector of regression coefficients that is robust against vertical outliers and leverage points. As a by-product, the method yields a weighted and imputed data set that contains estimates of what the values in cellwise outliers would need to amount to if they had fit the model. The method is illustrated to be equally robust as its casewise counterpart, MM regression. The cellwise regression method discards l…
Analyzing polytomous response from a complex survey scheme, like stratified or cluster sampling is very crucial in several socio-economics applications. We present a class of minimum quasi weighted density power divergence estimators for the polytomous logistic regression model with such a complex survey. This family of semiparametric estimators is a robust generalization of the maximum quasi weighted likelihood estimator exploiting the advantages of the popular density power divergence measure. Accordingly robust estimators for the design effects are also derived. Robust testing of general li…
Meta-analysis is an important statistical technique for synthesizing the results of multiple studies regarding the same or closely related research question. So-called meta-regression extends meta-analysis models by accounting for studylevel covariates. Mixed-effects meta-regression models provide a powerful tool for evidence synthesis, by appropriately accounting for betweem-study heterogeneity. In fact, modelling the study effect in terms of random effects and moderators not only allows to examine the impact of the moderators, but often leads to more accurate estimates of the involved parame…
Data on rates, percentages or proportions arise frequently in many different applied disciplines like medical biology, health care, psychology and several others. In this paper, we develop a robust inference procedure for the beta regression model which is used to describe such response variables taking values in $(0, 1)$ through some related explanatory variables. In relation to the beta regression model, the issue of robustness has been largely ignored in the literature so far. The existing maximum likelihood based inference has serious lack of robustness against outliers in data and generat…
The practical importance of inference with robustness against large bandwidths for causal effects in regression discontinuity and kink designs is widely recognized. Existing robust methods cover many cases, but do not handle uniform inference for CDF and quantile processes in fuzzy designs, despite its use in the recent literature in empirical microeconomics. In this light, this paper extends the literature by developing a unified framework of inference with robustness against large bandwidths that applies to uniform inference for quantile treatment effects in fuzzy designs, as well as all the…
We introduce flexible robust functional regression models, using various heavy-tailed processes, including a Student $t$-process. We propose efficient algorithms in estimating parameters for the marginal mean inferences and in predicting conditional means as well interpolation and extrapolation for the subject-specific inferences. We develop bootstrap prediction intervals for conditional mean curves. Numerical studies show that the proposed model provides robust analysis against data contamination or distribution misspecification, and the proposed prediction intervals maintain the nominal conf…
Fully robust versions of the elastic net estimator are introduced for linear and logistic regression. The algorithms to compute the estimators are based on the idea of repeatedly applying the non-robust classical estimators to data subsets only. It is shown how outlier-free subsets can be identified efficiently, and how appropriate tuning parameters for the elastic net penalties can be selected. A final reweighting step improves the efficiency of the estimators. Simulation studies compare with non-robust and other competing robust estimators and reveal the superiority of the newly proposed met…
Recently, there has been a growing need in analyzing data on manifolds owing to their important role in diverse fields of science and engineering. In the literature of manifold-valued data analysis up till now, however, only a few works have been carried out concerning the robustness of estimation against noises, outliers, and other sources of perturbations. In this regard, we introduce a novel extrinsic framework for analyzing manifold valued data in a robust manner. First, by extending the notion of the geometric median, we propose a new robust location parameter on manifolds, so-called the …
In linear regression, the least squares (LS) estimator has certain optimality properties if the errors are normally distributed. This assumption is often violated in practice, partly caused by data outliers. Robust estimators can cope with this situation and thus they are widely used in practice. One example of robust estimators for regression are adaptive modified maximum likelihood (AMML) estimators (Donmez, 2010). However, they are not robust to $x$ outliers, so-called leverage points. In this study, we propose a new regression estimator by employing an appropriate weighting scheme in the A…
This paper derives a new family of estimators, namely the minimum density power divergence estimators, as a robust generalization of the maximum likelihood estimator for the polytomous logistic regression model. Based on these estimators, a family of Wald-type test statistics for linear hypotheses is introduced. Robustness properties of both the proposed estimators and the test statistics are theoretically studied through the classical influence function analysis. Appropriate real life examples are presented to justify the requirement of suitable robust statistical procedures in place of the l…
We propose a robust variable selection procedure using a divergence based M-estimator combined with a penalty function. It produces robust estimates of the regression parameters and simultaneously selects the important explanatory variables. An efficient algorithm based on the quadratic approximation of the estimating equation is constructed. The asymptotic distribution and the influence function of the regression coefficients are derived. The widely used model selection procedures based on the Mallows's $C_p$ statistic and Akaike information criterion (AIC) often show very poor performance in…
Gradient boosting algorithms construct a regression predictor using a linear combination of ``base learners''. Boosting also offers an approach to obtaining robust non-parametric regression estimators that are scalable to applications with many explanatory variables. The robust boosting algorithm is based on a two-stage approach, similar to what is done for robust linear regression: it first minimizes a robust residual scale estimator, and then improves it by optimizing a bounded loss function. Unlike previous robust boosting proposals this approach does not require computing an ad-hoc residua…
