We obtain the upper error bounds of robust estimators for mean vector, using the median-of-means (MOM) method. The method is designed to handle data with heavy tails and contamination, with only a finite second moment, which is weaker than many others, relying on the VC dimension rather than the Rademacher complexity to measure statistical complexity. This allows us to implement MOM in covariance estimation, without imposing conditions such as $L$-sub-Gaussian or $L_{4}-L_{2}$ norm equivalence. In particular, we derive a new robust estimator, the MOM version of the halfspace depth, along with error bounds for mean estimation in any norm.
Robust estimators, like the median of a point set, are important for data analysis in the presence of outliers. We study robust estimators for locationally uncertain points with discrete distributions. That is, each point in a data set has a discrete probability distribution describing its location. The probabilistic nature of uncertain data makes it challenging to compute such estimators, since the true value of the estimator is now described by a distribution rather than a single point. We show how to construct and estimate the distribution of the median of a point set. Building the approxim…
In this note, we derive bounds on the median bias of univariate M-estimators under mild regularity conditions. These requirements are not sufficient to imply convergence in distribution of the M-estimators. We also discuss median bias of some multivariate M-estimators.
Tracking the Median of Gradients with a Stochastic Proximal Point Method
Technical University of Munich and Inria Paris, ENS, PSL Research University（慕尼黑技术大学和巴黎Inria、ENS、PSL研究大学）；Université Paris Cité and Sorbonne Université, CNRS（巴黎国立大学和索邦大学、CNRS）；Inria, CNRS, ENS, PSL Research University Paris（Inria、CNRS、ENS、PSL研究大学巴黎）；CCM, Flatiron Institute, Simons Foundation（CCM、Flatiron研究所、Simons基金会）
Efficient class of estimators for population median using auxiliary information
本文利用辅助变量给出了研究变量总体中位数的一类有效估计。得到了这类估计量的偏差和均方误差的渐近表达式。研究了渐近最优估计量及其近似均方误差。我们已经证明，提出的估计量比Srivastava（1967），Gross（1980），Kuk和Mak（1989）Singh et al.（2003 b），Al和Chingi（2009）以及Singh和Solanki（2013）考虑的估计量更有效。此外，理论研究结果支持的实证研究的基础上，两个人口显示的优越性，构建的估计比别人。
Notion of median in one dimension is a foundational element in nonparametric statistics. It has been extended to multi-dimensional cases both in location and in regression via notions of data depth. Regression depth (RD) and projection regression depth (PRD) represent the two most promising notions in regression. Carrizosa depth $D_C$ is another depth notion in regression.Depth induced regression medians (maximum depth estimators) serve as robust alternatives to the classical least squares estimator. The uniqueness of regression medians is indispensable in the discussion of their properties an…
Notions of depth in regression have been introduced and studied in the literature. Regression depth (RD) of Rousseeuw and Hubert (1999), the most famous one, is a direct extension of Tukey location depth (Tukey (1975)) to regression. Like its location counterpart, the most remarkable advantage of the notion of depth in regression is to directly introduce the maximum (or deepest) regression depth estimator (aka depth induced median) for regression parameters in a multi-dimensional setting. Classical questions for the regression depth induced median include (i) is it a consistent estimator (or r…
We introduce new estimators for robust machine learning based on median-of-means (MOM) estimators of the mean of real valued random variables. These estimators achieve optimal rates of convergence under minimal assumptions on the dataset. The dataset may also have been corrupted by outliers on which no assumption is granted. We also analyze these new estimators with standard tools from robust statistics. In particular, we revisit the concept of breakdown point. We modify the original definition by studying the number of outliers that a dataset can contain without deteriorating the estimation p…
We introduce a new notion of regularity of an estimator called median regularity. We prove that uniformly valid (honest) inference for a functional is possible if and only if there exists a median regular estimator of that functional. To our knowledge, such a notion of regularity that is necessary for uniformly valid inference is unavailable in the literature.
We consider median of means (MOM) versions of the Stahel-Donoho outlyingness (SDO) [stahel 1981, donoho 1982] and of Median Absolute Deviation (MAD) functions to construct subgaussian estimators of a mean vector under adversarial contamination and heavy-tailed data. We develop a single analysis of the MOM version of the SDO which covers all cases ranging from the Gaussian case to the L2 case. It is based on isomorphic and almost isometric properties of the MOM versions of SDO and MAD. This analysis also covers cases where the mean does not even exist but a location parameter does; in those cas…
In this paper, we introduce a robust nonparametric density estimator combining the popular Kernel Density Estimation method and the Median-of-Means principle (MoM-KDE). This estimator is shown to achieve robustness to any kind of anomalous data, even in the case of adversarial contamination. In particular, while previous works only prove consistency results under known contamination model, this work provides finite-sample high-probability error-bounds without a priori knowledge on the outliers. Finally, when compared with other robust kernel estimators, we show that MoM-KDE achieves competitiv…
Bias robustness of depth estimators in multivariate settings
FAMAF, National University of Córdoba（科尔多瓦国立大学 FAMAF）；CIEM and CONICET, Argentina（阿根廷 CIEM 和 CONICET）；Department of Mathematics, National University of Río Cuarto, Argentina（里奥夸尔托国立大学数学系）
Robust Nonparametric Methods for $α$-Mixing Sequences via Median-of-Means
HUN-REN Alfréd Rényi Institute of Mathematics（HUN-REN 阿尔弗雷德·雷尼数学研究所）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；HUN-REN Wigner Research Centre for Physics（HUN-REN 维格纳物理研究中心）
Let F_{θ} be a family of distributions with support on the set of nonnegative integers Z_0. In this paper we derive the M-estimators with smallest gross error sensitivity (GES). We start by defining the uniform median of a distribution F with support on Z_0 (umed(F)) as the median of x+u, where x and u are independent variables with distributions F and uniform in [-0.5,0.5] respectively. Under some general conditions we prove that the estimator with smallest GES satisfies umed(F_{n})=umed(F_{θ}), where F_{n} is the empirical distribution. The asymptotic distribution of these estimators is foun…
This paper addresses the following question: given a sample of i.i.d. random variables with finite variance, can one construct an estimator of the unknown mean that performs nearly as well as if the data were normally distributed? One of the most popular examples achieving this goal is the median of means estimator. However, it is inefficient in a sense that the constants in the resulting bounds are suboptimal. We show that a permutation-invariant modification of the median of means estimator admits deviation guarantees that are sharp up to $1+o(1)$ factor if the underlying distribution posses…
Median absolute deviation (hereafter MAD) is known as a robust alternative to the ordinary variance. It has been widely utilized to induce robust statistical inferential procedures. In this paper, we investigate the strong and weak Bahadur representations of its bootstrap counterpart. As a useful application, we utilize the results to derive the weak Bahadur representation of the bootstrap sample projection depth weighted mean---a quite important location estimator depending on MAD.
We analyze the performance of the Tukey median estimator under total variation (TV) distance corruptions. Previous results show that under Huber's additive corruption model, the breakdown point is 1/3 for high-dimensional halfspace-symmetric distributions. We show that under TV corruptions, the breakdown point reduces to 1/4 for the same set of distributions. We also show that a certain projection algorithm can attain the optimal breakdown point of 1/2. Both the Tukey median estimator and the projection algorithm achieve sample complexity linear in dimension.
The coefficient of variation (CV) is commonly used to measure relative dispersion. However, since it is based on the sample mean and standard deviation, outliers can adversely affect the CV. Additionally, for skewed distributions the mean and standard deviation do not have natural interpretations and, consequently, neither does the CV. Here we investigate the extent to which quantile-based measures of relative dispersion can provide appropriate summary information as an alternative to the CV. In particular, we investigate two measures, the first being the interquartile range (in lieu of the st…
Let $N_λ$ and $U$ be two independent random variables respectively distributed as a Poisson distribution with parameter $λ>0$ and a uniform distribution on $(0,1)$. This paper establishes that the median, say $M$, of $N_λ+U$ is close to $λ+1/3$ and more precisely that $M-λ-1/3=o(λ^{-1})$ as $λ\to \infty$. This result is used to construt a very simple robust estimator of $λ$ which is consistent and asymptotically normal. Compared to known robust estimates, this one can still be used with large datasets ($n\simeq 10^9$).
The sample mean is often used to aggregate different unbiased estimates of a parameter, producing a final estimate that is unbiased but possibly high-variance. This paper introduces the Bayesian median of means, an aggregation rule that roughly interpolates between the sample mean and median, resulting in estimates with much smaller variance at the expense of bias. While the procedure is non-parametric, its squared bias is asymptotically negligible relative to the variance, similar to maximum likelihood estimators. The Bayesian median of means is consistent, and concentration bounds for the es…
We propose a robust and scalable procedure for general optimization and inference problems on manifolds leveraging the classical idea of `median-of-means' estimation. This is motivated by ubiquitous examples and applications in modern data science in which a statistical learning problem can be cast as an optimization problem over manifolds. Being able to incorporate the underlying geometry for inference while addressing the need for robustness and scalability presents great challenges. We address these challenges by first proving a key lemma that characterizes some crucial properties of geomet…
We study the problem of estimating the mean of a random vector in $\mathbb{R}^d$ based on an i.i.d.\ sample, when the accuracy of the estimator is measured by a general norm on $\mathbb{R}^d$. We construct an estimator (that depends on the norm) that achieves an essentially optimal accuracy/confidence tradeoff under the only assumption that the random vector has a well-defined covariance matrix. The estimator is based on the construction of a uniform median-of-means estimator in a class of real valued functions that may be of independent interest.
We study the problem of estimating the mean of a random vector $X$ given a sample of $N$ independent, identically distributed points. We introduce a new estimator that achieves a purely sub-Gaussian performance under the only condition that the second moment of $X$ exists. The estimator is based on a novel concept of a multivariate median.
This work is concerned with the estimation of the intensity parameter of a stationary determinantal point process. We consider the standard estimator, corresponding to the number of observed points per unit volume and a recently introduced median-based estimator more robust to outliers. The consistency and asymptotic normality of estimators are obtained under mild assumptions on the determinantal point process. We illustrate the efficiency of the procedures in a simulation study.
We generalize the naïve estimator of a Poisson regression model with measurement errors as discussed in Kukush et al. [1]. The explanatory variable is not always normally distributed as they assume. In this study, we assume that the explanatory variable and measurement error are not limited to a normal distribution. We clarify the requirements for the existence of the naïve estimator and derive its asymptotic bias and asymptotic mean squared error (MSE). In addition, we propose a consistent estimator of the true parameter by correcting the bias of the naïve estimator. As illustrative examples,…
We identify and correct an error in the paper "Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?" This correction allows new guarantees on the excess degrees of freedom--the bias in the error estimate of Stein's unbiased risk estimate (SURE) for an estimator tuned by directly minimizing the SURE criterion--for arbitrary SURE-tuned linear estimators. Oracle inequalities follow as a consequence of these results for such estimators.
In this paper, we compare maximum likelihood (ML), quasi likelihood (QL) and weighted least squares (WLS) estimators for proportional error nonlinear regression models. Literature on thermoluminescence sedimentary dating revealed another estimator similar to weighted least squares but observed responses used as weights. This estimator that we refer to as data weighted least squares (DWLS) is also included in the comparison. We show that on the order $σ, $ all four estimators behave similar to ordinary least squares estimators for standard linear regression models. On the order of $σ^2, $ the e…
This paper discusses minimum distance estimation method in the linear regression model with dependent errors which are strongly mixing. The regression parameters are estimated through the minimum distance estimation method, and asymptotic distributional properties of the estimators are discussed. A simulation study compares the performance of the minimum distance estimator with other well celebrated estimator. This simulation study shows the superiority of the minimum distance estimator over another estimator. KoulMde (R package) which was used for the simulation study is available online. See…
Convex Clustering Redefined: Robust Learning with the Median of Means Estimator
Indian Statistical Institute, Kolkata（印度统计研究院，科希马）；Department of Statistics and Data Science, University of Texas at Austin（德克萨斯大学奥斯汀分校统计与数据科学系）；Electronics and Communication Sciences, Indian Statistical Institute, Kolkata（印度统计研究院，科希马电子与通信科学）；Department of Statistics, University of Oxford（牛津大学统计系）；Department of Statistics, University of Michigan（密歇根大学统计系）
This paper considers the penalized least squares estimator with arbitrary convex penalty. When the observation noise is Gaussian, we show that the prediction error is a subgaussian random variable concentrated around its median. We apply this concentration property to derive sharp oracle inequalities for the prediction error of the LASSO, the group LASSO and the SLOPE estimators, both in probability and in expectation. In contrast to the previous work on the LASSO type methods, our oracle inequalities in probability are obtained at any confidence level for estimators with tuning parameters tha…
We study the least square estimator, in the framework of simple linear regression, when the deviance term $\varepsilon$ with respect to the linear model is modeled by a uniform distribution. In particular, we give the law of this estimator, and prove some convergence properties.
