We present a new finite-sample analysis of M-estimators of locations in $\mathbb{R}^d$ using the tool of the influence function. In particular, we show that the deviations of an M-estimator can be controlled thanks to its influence function (or its score function) and then, we use concentration inequality on M-estimators to investigate the robust estimation of the mean in high dimension in a corrupted setting (adversarial corruption setting) for bounded and unbounded score functions. For a sample of size $n$ and covariance matrix $Σ$, we attain the minimax speed $\sqrt{Tr(Σ)/n}+\sqrt{\|Σ\|_{op}\log(1/δ)/n}$ with probability larger than $1-δ$ in a heavy-tailed setting. One of the major advantages of our approach compared to others recently proposed is that our estimator is tractable and fast to compute even in very high dimension with a complexity of $O(nd\log(Tr(Σ)))$ where $n$ is the sample size and $Σ$ is the covariance matrix of the inliers. In practice, the code that we make available for this article proves to be very fast.
Motivated by recent data analytics applications, we study the adversarial robustness of robust estimators. Instead of assuming that only a fraction of the data points are outliers as considered in the classic robust estimation setup, in this paper, we consider an adversarial setup in which an attacker can observe the whole dataset and can modify all data samples in an adversarial manner so as to maximize the estimation error caused by his attack. We characterize the attacker's optimal attack strategy, and further introduce adversarial influence function (AIF) to quantify an estimator's sensiti…
Consider the standard nonparametric regression model and take as estimator the penalized least squares function. In this article, we study the trade-off between closeness to the true function and complexity penalization of the estimator, where complexity is described by a seminorm on a class of functions. First, we present an exponential concentration inequality revealing the concentration behavior of the trade-off of the penalized least squares estimator around a nonrandom quantity, where such quantity depends on the problem under consideration. Then, under some conditions and for the proper …
This article studies the \emph{robust covariance matrix estimation} of a data collection $X = (x_1,\ldots,x_n)$ with $x_i = \sqrt τ_i z_i + m$, where $z_i \in \mathbb R^p$ is a \textit{concentrated vector} (e.g., an elliptical random vector), $m\in \mathbb R^p$ a deterministic signal and $τ_i\in \mathbb R$ a scalar perturbation of possibly large amplitude, under the assumption where both $n$ and $p$ are large. This estimator is defined as the fixed point of a function which we show is contracting for a so-called \textit{stable semi-metric}. We exploit this semi-metric along with concentration …
We consider the horseshoe estimator due to Carvalho, Polson and Scott (2010) for the multivariate normal mean model in the situation that the mean vector is sparse in the nearly black sense. We assume the frequentist framework where the data is generated according to a fixed mean vector. We show that if the number of nonzero parameters of the mean vector is known, the horseshoe estimator attains the minimax $\ell_2$ risk, possibly up to a multiplicative constant. We provide conditions under which the horseshoe estimator combined with an empirical Bayes estimate of the number of nonzero means s…
We analyze a plug-in estimator for a large class of integral functionals of one or more continuous probability densities. This class includes important families of entropy, divergence, mutual information, and their conditional versions. For densities on the $d$-dimensional unit cube $[0,1]^d$ that lie in a $β$-Hölder smoothness class, we prove our estimator converges at the rate $O \left( n^{-\fracβ{β+ d}} \right)$. Furthermore, we prove the estimator is exponentially concentrated about its mean, whereas most previous related results have proven only expected error bounds on estimators.
This paper presents an adaptive version of the Hill estimator based on Lespki's model selection method. This simple data-driven index selection method is shown to satisfy an oracle inequality and is checked to achieve the lower bound recently derived by Carpentier and Kim. In order to establish the oracle inequality, we derive non-asymptotic variance bounds and concentration inequalities for Hill estimators. These concentration inequalities are derived from Talagrand's concentration inequality for smooth functions of independent exponentially distributed random variables combined with three to…
The Maximum Likelihood Estimator (MLE) serves an important role in statistics and machine learning. In this article, for i.i.d. variables, we obtain constant-specified and sharp concentration inequalities and oracle inequalities for the MLE only under exponential moment conditions. Furthermore, in a robust setting, the sub-Gaussian type oracle inequalities of the log-truncated maximum likelihood estimator are derived under the second-moment condition.
In this paper, we study a class of non-parametric density estimators under Bayesian settings. The estimators are piecewise constant functions on binary partitions. We analyze the concentration rate of the posterior distribution under a suitable prior, and demonstrate that the rate does not directly depend on the dimension of the problem. This paper can be viewed as an extension of a parallel work where the convergence rate of a related sieve MLE was established. Compared to the sieve MLE, the main advantage of the Bayesian method is that it can adapt to the unknown complexity of the true densi…
This paper considers the penalized least squares estimator with arbitrary convex penalty. When the observation noise is Gaussian, we show that the prediction error is a subgaussian random variable concentrated around its median. We apply this concentration property to derive sharp oracle inequalities for the prediction error of the LASSO, the group LASSO and the SLOPE estimators, both in probability and in expectation. In contrast to the previous work on the LASSO type methods, our oracle inequalities in probability are obtained at any confidence level for estimators with tuning parameters tha…
Our goal is to establish large deviations and concentration inequalities for the maximum likelihood estimator of the drift parameter of the Ornstein-Uhlenbeck process without tears. We propose a new strategy to establish large deviation results which allows us, via a suitable transformation, to circumvent the classical difficulty of non-steepness. Our approach holds in the stable case where the process is positive recurrent as well as in the unstable and explosive cases where the process is respectively null recurrent and transient. Notwithstanding of this trichotomy, we also provide new conce…
By deriving influence functions related to multiple-set linear canonical analysis (MSLCA) we show that the classical version of this analysis, based on empirical covariance operators, is not robust. Then, we introduce a robust version of MSLCA by using the MCD estimator of the covariance operator of the involved random vector. The related influence functions are then derived and are shown to be bounded. Asymptotic properties of the introduced robust MSLCA are obtained and permit to propose a robust test for mutual non-correlation. This test is shown to be robust by studying the related second …
We study the estimation and concentration on its expectation of the probability to observe data further than a specified distance from a given iid sample in a metric space. The problem extends the classical problem of estimation of the missing mass in discrete spaces. We give some estimators for the conditional missing mass and show that estimation of the expected missing mass is difficult in general. Conditions on the distribution, under which the Good-Turing estimator and the conditional missing mass concentrate on their expectations are identified. Applications to anomaly detection, coding,…
We propose a general framework for regularization in M-estimation problems under time dependent (absolutely regular-mixing) data which encompasses many of the existing estimators. We derive non-asymptotic concentration bounds for the regularized M-estimator. Our results exhibit a variance-bias trade-off, with the variance term being governed by a novel measure of the complexity of the parameter set. We also show that the mixing structure affect the variance term by scaling the number of observations; depending on the decay rate of the mixing coefficients, this scaling can even affect the asymp…
This work examines the problem of graph learning over a diffusion network when data can be collected from a limited portion of the network (partial observability). The main question is to establish technical guarantees of consistent recovery of the subgraph of probed network nodes, i) despite the presence of unobserved nodes; and ii) under different connectivity regimes, including the dense regime where the probed nodes are influenced by many connections coming from the unobserved ones. We ascertain that suitable estimators of the combination matrix (i.e., the matrix that quantifies the pairwi…
Evaluation of treatment effects and more general estimands is typically achieved via parametric modelling, which is unsatisfactory since model misspecification is likely. Data-adaptive model building (e.g. statistical/machine learning) is commonly employed to reduce the risk of misspecification. Naive use of such methods, however, delivers estimators whose bias may shrink too slowly with sample size for inferential methods to perform well, including those based on the bootstrap. Bias arises because standard data-adaptive methods are tuned towards minimal prediction error as opposed to e.g. min…
Semiparametric Efficient Empirical Higher Order Influence Function Estimators
Institute of Natural Sciences, MOE-LSC, School of Mathematical Sciences, CMA-Shanghai and SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University（上海交通大学自然学院、教育部数学科学中心、CMA-上海和上海交通大学-耶鲁生物统计与数据科学联合中心）；Department of Biostatistics, Harvard University（哈佛大学生物统计系）；Department of Economics, Massachusetts Institute of Technology（麻省理工学院经济学系）；Department of Epidemiology and Biostatistics, Harvard University（哈佛大学流行病学与生物统计系）
Optimal concentration and gradient sensing by perfect monitors
Institute for Theoretical Physics, Department of Physics, Utrecht University（乌得勒支大学理论物理系）；Centre for Complex Systems Studies, Utrecht University（乌得勒支大学复杂系统研究中心）
The precision matrix that encodes conditional linear dependency relations among a set of variables forms an important object of interest in multivariate analysis. Sparse estimation procedures for precision matrices such as the graphical lasso (Glasso) gained popularity as they facilitate interpretability, thereby separating pairs of variables that are conditionally dependent from those that are independent (given all other variables). Glasso lacks, however, robustness to outliers. To overcome this problem, one typically applies a robust plug-in procedure where the Glasso is computed from a rob…
Estimators based on influence functions (IFs) have been shown to be effective in many settings, especially when combined with machine learning techniques. By focusing on estimating a specific target of interest (e.g., the average effect of a treatment), rather than on estimating the full underlying data generating distribution, IF-based estimators are often able to achieve asymptotically optimal mean-squared error. Still, many researchers find IF-based estimators to be opaque or overly technical, which makes their use less prevalent and their benefits less available. To help foster understandi…
In this paper we study the properties of the Lasso estimator of the drift component in the diffusion setting. More specifically, we consider a multivariate parametric diffusion model $X$ observed continuously over the interval $[0,T]$ and investigate drift estimation under sparsity constraints. We allow the dimensions of the model and the parameter space to be large. We obtain an oracle inequality for the Lasso estimator and derive an error bound for the $L^2$-distance using concentration inequalities for linear functionals of diffusion processes. The probabilistic part is based upon elements …
Dalhousie University（达尔豪斯大学）；University of Science（理大学）；Vietnam National University（越南国立大学）；AISIA Research Lab（AISIA研究实验室）；University of Delaware（特拉华大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
In this paper, we study the learning rate of generalized Bayes estimators in a general setting where the hypothesis class can be uncountable and have an irregular shape, the loss function can have heavy tails, and the optimal hypothesis may not be unique. We prove that under the multi-scale Bernstein's condition, the generalized posterior distribution concentrates around the set of optimal hypotheses and the generalized Bayes estimator can achieve fast learning rate. Our results are applied to show that the standard Bayesian linear regression is robust to heavy-tailed distributions.
This paper aims to provide a tutorial for upper level undergraduate and graduate students in statistics, biostatistics and epidemiology on deriving influence functions for non-parametric and semi-parametric models. The author will build on previously known efficiency theory and provide a useful identity and formulaic technique only relying on the basics of integration which, are self-contained in this tutorial and can be used in most any setting one might encounter in practice. The paper provides many examples of such derivations for well-known influence functions as well as for new parameters…
Estimating divergences in a consistent way is of great importance in many machine learning tasks. Although this is a fundamental problem in nonparametric statistics, to the best of our knowledge there has been no finite sample exponential inequality convergence bound derived for any divergence estimators. The main contribution of our work is to provide such a bound for an estimator of Rényi-$α$ divergence for a smooth Hölder class of densities on the $d$-dimensional unit cube $[0, 1]^d$. We also illustrate our theoretical results with a numerical experiment.
We generalize the { ${\rm M}$-estimator} put forward by Catoni in his seminal paper [C12] to the case in which samples can have finite $α$-th moment with $α\in (1,2)$ rather than finite variance, our approach is by slightly modifying the influence function $φ$ therein. The choice of the new influence function is inspired by the Taylor-like expansion developed in [C-N-X]. We obtain a deviation bound of the estimator, as $α\rightarrow 2$, this bound is the same as that in [C12]. Experiment shows that our generalized ${\rm M}$-estimator performs better than the empirical mean estimator, the small…
Higher-Order Influence Functions (HOIFs) provide a unified theory for constructing rate-optimal estimators for a large class of low-dimensional (smooth) statistical functionals/parameters (and sometimes even infinite-dimensional functions) that arise in substantive fields including epidemiology, economics, and the social sciences. Since the introduction of HOIFs by Robins et al. (2008), they have been viewed mostly as a theoretical benchmark rather than a useful tool for statistical practice. Works aimed to flip the script are scant, but a few recent papers Liu et al. (2017, 2021b) make some p…
We consider a robust version of the classical Wald test statistics for testing simple and composite null hypotheses for general parametric models. These test statistics are based on the minimum density power divergence estimators instead of the maximum likelihood estimators. An extensive study of their robustness properties is given though the influence functions as well as the chi-square inflation factors. It is theoretically established that the level and power of these robust tests are stable against outliers, whereas the classical Wald test breaks down. Some numerical examples confirm the …
Stabilized Higher-Order Influence Functions: Statistical Theory of a Class of Bilinear Forms
School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学科学学院）；Department of Statistics, University of Virginia（弗吉尼亚大学统计学系）；Department of Statistics and Data Science, Tsinghua University（清华大学统计与数据科学系）；Institute of Natural Sciences, MOE–LSC, CMA–Shanghai, SJTU–Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University（上海交通大学自然科学研究院、教育部数学天元基金管理中心-上海数学中心、上海交通大学-耶鲁大学生物统计与数据科学联合中心）
Multiply Robust Causal Mediation Analysis with Continuous Treatments
Division of Biostatistics, Department of Population Health Sciences, University of Utah（犹他大学人口健康科学系生物统计部）；Department of Mathematics and Statistics, Boston University（波士顿大学数学与统计系）；Department of Computer Science, Johns Hopkins University（约翰霍普金斯大学计算机科学系）
Estimating entropy and mutual information consistently is important for many machine learning applications. The Kozachenko-Leonenko (KL) estimator (Kozachenko & Leonenko, 1987) is a widely used nonparametric estimator for the entropy of multivariate continuous random variables, as well as the basis of the mutual information estimator of Kraskov et al. (2004), perhaps the most widely used estimator of mutual information in this setting. Despite the practical importance of these estimators, major theoretical questions regarding their finite-sample behavior remain open. This paper proves finite-s…
