Notions of depth in regression have been introduced and studied in the literature. Regression depth (RD) of Rousseeuw and Hubert (1999), the most famous one, is a direct extension of Tukey location depth (Tukey (1975)) to regression.
Like its location counterpart, the most remarkable advantage of the notion of depth in regression is to directly introduce the maximum (or deepest) regression depth estimator (aka depth induced median) for regression parameters in a multi-dimensional setting. Classical questions for the regression depth induced median include (i) is it a consistent estimator (or rather under what sufficient conditions, it is consistent)? and (ii) is there any limiting distribution?
Bai and He (1999) (BH99) pioneered an attempt to answer these questions. Under some stringent conditions on (i) the design points, (ii) the conditional distributions of $y$ given $\bs{x}_i$, and (iii) the error distributions, BH99 proved the strong consistency of the depth induced median. Under another set of conditions, BH99 showed the existence of the limiting distribution of the estimator.
This article establishes the strong consistency of the depth induced median without any of the stringent conditions in BH99, and proves the existence of the limiting distribution of the estimator by sufficient conditions and an approach different from BH99.
Notion of median in one dimension is a foundational element in nonparametric statistics. It has been extended to multi-dimensional cases both in location and in regression via notions of data depth. Regression depth (RD) and projection regression depth (PRD) represent the two most promising notions in regression. Carrizosa depth $D_C$ is another depth notion in regression.Depth induced regression medians (maximum depth estimators) serve as robust alternatives to the classical least squares estimator. The uniqueness of regression medians is indispensable in the discussion of their properties an…
Median absolute deviation (hereafter MAD) is known as a robust alternative to the ordinary variance. It has been widely utilized to induce robust statistical inferential procedures. In this paper, we investigate the strong and weak Bahadur representations of its bootstrap counterpart. As a useful application, we utilize the results to derive the weak Bahadur representation of the bootstrap sample projection depth weighted mean---a quite important location estimator depending on MAD.
In this paper we develop a nonparametric regression method that is simultaneously adaptive over a wide range of function classes for the regression function and robust over a large collection of error distributions, including those that are heavy-tailed, and may not even possess variances or means. Our approach is to first use local medians to turn the problem of nonparametric regression with unknown noise distribution into a standard Gaussian regression problem and then apply a wavelet block thresholding procedure to construct an estimator of the regression function. It is shown that the esti…
In this paper, a nonparametric estimator is proposed for estimating the L1-median for multivariate conditional distribution when the covariates take values in an infinite dimensional space. The multivariate case is more appropriate to predict the components of a vector of random variables simultaneously rather than predicting each of them separately. While estimating the conditional L1-median function using the well-known Nadarya-Waston estimator, we establish the strong consistency of this estimator as well as the asymptotic normality. We also present some simulations and provide how to built…
This article is devoted to the problem of predicting the value taken by a random permutation $Σ$, describing the preferences of an individual over a set of numbered items $\{1,\; \ldots,\; n\}$ say, based on the observation of an input/explanatory r.v. $X$ e.g. characteristics of the individual), when error is measured by the Kendall $τ$ distance. In the probabilistic formulation of the 'Learning to Order' problem we propose, which extends the framework for statistical Kemeny ranking aggregation developped in \citet{CKS17}, this boils down to recovering conditional Kemeny medians of $Σ$ given …
For any given partial order in a $d$-dimensional euclidean space, under mild regularity assumptions, we show that the intersection of closed (generalized) intervals containing more than 1/2 of the probability mass, is a non-empty compact interval. This property is shared with common intervals on real line, where the intersection is the median set of the underlying probability distribution. So obtained multivariate medians with respect to a partial order, can be observed as special cases of centers of distribution in the sense of type D depth functions introduced by Y. Zuo and R. Serfling, {\em…
For multivariate data, Tukey's half-space depth is one of the most popular depth functions available in the literature. It is conceptually simple and satisfies several desirable properties of depth functions. The Tukey median, the multivariate median associated with the half-space depth, is also a well-known measure of center for multivariate data with several interesting properties. In this article, we derive and investigate some interesting properties of half-space depth and its associated multivariate median. These properties, some of which are counterintuitive, have important statistical c…
In this paper, we investigate the differentially private estimation of data depth functions and their associated medians. We introduce several methods for privatizing depth values at a fixed point, and show that for some depth functions, when the depth is computed at an out of sample point, privacy can be gained for free when $n\rightarrow \infty$. We also present a method for privately estimating the vector of sample point depth values. Additionally, we introduce estimation methods for depth-based medians for both depth functions with low global sensitivity and depth functions with only highl…
Identification of the center of a data cloud is one of the basic problems in statistics. One popular choice for such a center is the median, and several versions of median in finite dimensional spaces have been studied in the literature. In particular, medians based on different notions of data depth have been extensively studied by many researchers, who defined median as the point, where the depth function attains its maximum value. In other words, the median is the deepest point in the sample space according to that definition. In this paper, we investigate the deepest point for probability …
The halfspace depth is a prominent tool of nonparametric multivariate analysis. The upper level sets of the depth, termed the trimmed regions of a measure, serve as a natural generalization of the quantiles and inter-quantile regions to higher-dimensional spaces. The smallest non-empty trimmed region, coined the halfspace median of a measure, generalizes the median. We focus on the (inverse) ray basis theorem for the halfspace depth, a crucial theoretical result that characterizes the halfspace median by a covering property. First, a novel elementary proof of that statement is provided, under …
Discussion of "Multivariate quantiles and multiple-output regression quantiles: From $L_1$ optimization to halfspace depth" by M. Hallin, D. Paindaveine and M. Siman [arXiv:1002.4486]
Discussion of "Multivariate quantiles and multiple-output regression quantiles: From $L_1$ optimization to halfspace depth" by M. Hallin, D. Paindaveine and M. Siman [arXiv:1002.4486]
Discussion of "Multivariate quantiles and multiple-output regression quantiles: From $L_1$ optimization to halfspace depth" by M. Hallin, D. Paindaveine and M. Siman [arXiv:1002.4486]
This paper studies robust regression in the settings of Huber's $ε$-contamination models. We consider estimators that are maximizers of multivariate regression depth functions. These estimators are shown to achieve minimax rates in the settings of $ε$-contamination models for various regression problems including nonparametric regression, sparse linear regression, reduced rank regression, etc. We also discuss a general notion of depth function for linear operators that has potential applications in robust functional linear regression.
Rejoinder to "Multivariate quantiles and multiple-output regression quantiles: From $L_1$ optimization to halfspace depth" by M. Hallin, D. Paindaveine and M. Siman [arXiv:1002.4486]
We investigate asymptotic properties of least-absolute-deviation or median quantile estimates of the location and scale functions in nonparametric regression models with dependent data from multiple subjects. Under a general dependence structure that allows for longitudinal data and some spatially correlated data, we establish uniform Bahadur representations for the proposed median quantile estimates. The obtained Bahadur representations provide deep insights into the asymptotic behavior of the estimates. Our main theoretical development is based on studying the modulus of continuity of kernel…
We consider the non-parametric regression problem under Huber's $ε$-contamination model, in which an $ε$ fraction of observations are subject to arbitrary adversarial noise. We first show that a simple local binning median step can effectively remove the adversary noise and this median estimator is minimax optimal up to absolute constants over the Hölder function class with smoothness parameters smaller than or equal to 1. Furthermore, when the underlying function has higher smoothness, we show that using local binning median as pre-preprocessing step to remove the adversarial noise, then we c…
Bias robustness of depth estimators in multivariate settings
FAMAF, National University of Córdoba（科尔多瓦国立大学 FAMAF）；CIEM and CONICET, Argentina（阿根廷 CIEM 和 CONICET）；Department of Mathematics, National University of Río Cuarto, Argentina（里奥夸尔托国立大学数学系）
Despite the renewed interest in the Newey and Powell (1987) concept of expectiles in fields such as econometrics, risk management, and extreme value theory, expectile regression---or, more generally, M-quantile regression---unfortunately remains limited to single-output problems. To improve on this, we introduce hyperplane-valued multivariate M-quantiles that show strong advantages, for instance in terms of equivariance, over the various point-valued multivariate M-quantiles available in the literature. Like their competitors, our multivariate M-quantiles are directional in nature and provide …
We develop uniformly valid confidence regions for regression coefficients in a high-dimensional sparse median regression model with homoscedastic errors. Our methods are based on a moment equation that is immunized against non-regular estimation of the nuisance part of the median regression function by using Neyman's orthogonalization. We establish that the resulting instrumental median regression estimator of a target regression coefficient is asymptotically normally distributed uniformly with respect to the underlying sparse model and is semi-parametrically efficient. We also generalize our …
We consider the classical statistical learning/regression problem, when the value of a real random variable Y is to be predicted based on the observation of another random variable X. Given a class of functions F and a sample of independent copies of (X, Y ), one needs to choose a function f from F such that f(X) approximates Y as well as possible, in the mean-squared sense. We introduce a new procedure, the so-called median-of-means tournament, that achieves the optimal tradeoff between accuracy and confidence under minimal assumptions, and in particular outperforms classical methods based on…
A new multivariate concept of quantile, based on a directional version of Koenker and Bassett's traditional regression quantiles, is introduced for multivariate location and multiple-output regression problems. In their empirical version, those quantiles can be computed efficiently via linear programming techniques. Consistency, Bahadur representation and asymptotic normality results are established. Most importantly, the contours generated by those quantiles are shown to coincide with the classical halfspace depth contours associated with the name of Tukey. This relation does not only allow f…
Under special conditions on data set and underlying distribution, the limit of finite sample breakdown point of Tukey's halfspace median ($\frac{1} {3}$) has been obtained in literature. In this paper, we establish the result under \emph{weaker assumption} imposed on underlying distribution (halfspace symmetry) and on data set (not necessary in general position). The representation of Tukey's sample depth regions for data set \emph{not necessary in general position} is also obtained, as a by-product of our derivation.
A regularized risk minimization procedure for regression function estimation is introduced that achieves near optimal accuracy and confidence under general conditions, including heavy-tailed predictor and response variables. The procedure is based on median-of-means tournaments, introduced by the authors in [8]. It is shown that the new procedure outperforms standard regularized empirical risk minimization procedures such as lasso or slope in heavy-tailed problems.
A new quantile regression concept, based on a directional version of Koenker and Bassett's traditional single-output one, has been introduced in [Ann. Statist. (2010) 38 635-669] for multiple-output location/linear regression problems. The polyhedral contours provided by the empirical counterpart of that concept, however, cannot adapt to unknown nonlinear and/or heteroskedastic dependencies. This paper therefore introduces local constant and local linear (actually, bilinear) versions of those contours, which both allow to asymptotically recover the conditional halfspace depth contours that com…
A nonparametric procedure for robust regression estimation and for quantile regression is proposed which is completely data-driven and adapts locally to the regularity of the regression function. This is achieved by considering in each point M-estimators over different local neighbourhoods and by a local model selection procedure based on sequential testing. Non-asymptotic risk bounds are obtained, which yield rate-optimality for large sample asymptotics under weak conditions. Simulations for different univariate median regression models show good finite sample properties, also in comparison t…
In 1975 John Tukey proposed a multivariate median which is the 'deepest' point in a given data cloud in R^d. Later, in measuring the depth of an arbitrary point z with respect to the data, David Donoho and Miriam Gasko considered hyperplanes through z and determined its 'depth' by the smallest portion of data that are separated by such a hyperplane. Since then, these ideas has proved extremely fruitful. A rich statistical methodology has developed that is based on data depth and, more general, nonparametric depth statistics. General notions of data depth have been introduced as well as many sp…
Data depth functions are a generalization of one-dimensional order statistics and medians to real spaces of dimension greater than one; in particular, a data depth function quantifies the centrality of a point with respect to a data set or a probability distribution. One of the most commonly studied data depth functions is halfspace depth. It is of interest to computational geometers because it is highly geometric, and it is of interest to statisticians because it shares many desirable theoretical properties with the one-dimensional median. As the sample size increases, the halfspace depth for…
In many real-world applications, collected data are contaminated by noise with heavy-tailed distribution and might contain outliers of large magnitude. In this situation, it is necessary to apply methods which produce reliable outcomes even if the input contains corrupted measurements. We describe a general method which allows one to obtain estimators with tight concentration around the true parameter of interest taking values in a Banach space. Suggested construction relies on the fact that the geometric median of a collection of independent "weakly concentrated" estimators satisfies a much s…
Depth notions in regression have been systematically proposed and examined in Zuo (2018). One of the prominent advantages of notion of depth is that it can be directly utilized to introduce median-type deepest estimating functionals (or estimators in empirical distribution case) for location or regression parameters in a multi-dimensional setting. Regression depth shares the advantage. Depth induced deepest estimating functionals are expected to inherit desirable and inherent robustness properties ( e.g. bounded maximum bias and influence function and high breakdown point) as their univariate …
We characterize the exponential distribution as the only one which satisfies a regression condition. This condition involves the regression function of a fixed record value given two other record values, one of them being previous and the other next to the fixed record value, and none of them are adjacent. In particular, it turns out that the underlying distribution is exponential if and only if given the first and last record values, the expected value of the median in a sample of record values equals the sample midrange.
In this paper, we propose a class of high breakdown point estimators for the linear regression model when the response variable contains censored observations. These estimators are robust against high-leverage outliers and they generalize the LMS (least median of squares), S, MM and $τ$-estimators for linear regression. An important contribution of this paper is that we can define consistent estimators using a bounded loss function (or equivalently, a redescending score function). Since the calculation of these estimators can be computationally costly, we propose an efficient algorithm to comp…
Depth of the Tukey median is investigated for empirical distributions. A sharper upper bound is provided for this value for data sets in general position. This bound is lower than the existing one in the literature, and more importantly derived under the \emph{fixed} sample size practical scenario. Several results obtained in this paper are interesting theoretically and useful as well to reduce the computational burden of the Tukey median practically when $p$ is large relative to large $n$.
Pompeu Fabra University（庞培法布拉大学）；ICREA（加泰罗尼亚研究与高等研究院）；Barcelona Graduate School of Economics（巴塞罗那经济研究生院）；The Australian National University（澳大利亚国立大学）；Sorbonne University（索邦大学）
We survey some of the recent advances in mean estimation and regression function estimation. In particular, we describe sub-Gaussian mean estimators for possibly heavy-tailed data both in the univariate and multivariate settings. We focus on estimators based on median-of-means techniques but other methods such as the trimmed mean and Catoni's estimator are also reviewed. We give detailed proofs for the cornerstone results. We dedicate a section on statistical learning problems--in particular, regression function estimation--in the presence of possibly heavy-tailed data.
Robust Nonparametric Methods for $α$-Mixing Sequences via Median-of-Means
HUN-REN Alfréd Rényi Institute of Mathematics（HUN-REN 阿尔弗雷德·雷尼数学研究所）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；HUN-REN Wigner Research Centre for Physics（HUN-REN 维格纳物理研究中心）
