We provide a detailed study of the estimation of probability distributions---discrete and continuous---in a stringent setting in which data is kept private even from the statistician. We give sharp minimax rates of convergence for estimation in these locally private settings, exhibiting fundamental tradeoffs between privacy and convergence rate, as well as providing tools to allow movement along the privacy-statistical efficiency continuum. One of the consequences of our results is that Warner's classical work on randomized response is an optimal way to perform survey sampling while maintaining privacy of the respondents.
Working under a model of privacy in which data remains private even from the statistician, we study the tradeoff between privacy guarantees and the utility of the resulting statistical estimators. We prove bounds on information-theoretic quantities, including mutual information and Kullback-Leibler divergence, that depend on the privacy guarantees. When combined with standard minimax techniques, including the Le Cam, Fano, and Assouad methods, these inequalities allow for a precise characterization of statistical rates under local privacy constraints. We provide a treatment of several canonica…
We explore and compare a variety of definitions for privacy and disclosure limitation in statistical estimation and data analysis, including (approximate) differential privacy, testing-based definitions of privacy, and posterior guarantees on disclosure risk. We give equivalence results between the definitions, shedding light on the relationships between different formalisms for privacy. We also take an inferential perspective, where---building off of these definitions---we provide minimax risk bounds for several estimation problems, including mean estimation, estimation of the support of a di…
University of Pennsylvania（宾夕法尼亚大学）；Rutgers University（罗格斯大学）
We propose differentially private algorithms for parameter estimation in both low-dimensional and high-dimensional sparse generalized linear models (GLMs) by constructing private versions of projected gradient descent. We show that the proposed algorithms are nearly rate-optimal by characterizing their statistical performance and establishing privacy-constrained minimax lower bounds for GLMs. The lower bounds are obtained via a novel technique, which is based on Stein's Lemma and generalizes the tracing attack technique for privacy-constrained lower bounds. This lower bound argument can be of …
Estimation problems with constrained parameter spaces arise in various settings. In many of these problems, the observations available to the statistician can be modelled as arising from the noisy realization of the image of a random linear operator; an important special case is random design regression. We derive sharp rates of estimation for arbitrary compact elliptical parameter sets and demonstrate how they depend on the distribution of the random linear operator. Our main result is a functional that characterizes the minimax rate of estimation in terms of the noise level, the law of the r…
Working under a model of privacy in which data remains private even from the statistician, we study the tradeoff between privacy guarantees and the risk of the resulting statistical estimators. We develop private versions of classical information-theoretic bounds, in particular those due to Le Cam, Fano, and Assouad. These inequalities allow for a precise characterization of statistical rates under local privacy constraints and the development of provably (minimax) optimal estimation procedures. We provide a treatment of several canonical families of problems: mean estimation and median estima…
Finding anonymization mechanisms to protect personal data is at the heart of recent machine learning research. Here, we consider the consequences of local differential privacy constraints on goodness-of-fit testing, i.e. the statistical problem assessing whether sample points are generated from a fixed density $f_0$, or not. The observations are kept hidden and replaced by a stochastic transformation satisfying the local differential privacy constraint. In this setting, we propose a testing procedure which is based on an estimation of the quadratic distance between the density $f$ of the unobs…
We consider the privacy problem of statistical estimation from distributed data, where users communicate to a central processor over a Gaussian multiple-access channel(MAC). To avoid the inevitable sacrifice of data utility for privacy in digital transmission schemes, we devise an over-the-air estimation strategy which utilizes the additive nature of MAC channel. Using the mutual information between the channel outputs and users' data as the metric, we obtain the privacy bounds for our scheme and validate that it can guarantee strong privacy without incurring larger estimation error. Further, …
We consider the goodness-of fit testing problem for Hölder smooth densities over $\mathbb{R}^d$: given $n$ iid observations with unknown density $p$ and given a known density $p_0$, we investigate how large $ρ$ should be to distinguish, with high probability, the case $p=p_0$ from the composite alternative of all Hölder-smooth densities $p$ such that $\|p-p_0\|_t \geq ρ$ where $t \in [1,2]$. The densities are assumed to be defined over $\mathbb{R}^d$ and to have Hölder smoothness parameter $α>0$. In the present work, we solve the case $α\leq 1$ and handle the case $α>1$ using an additional tec…
We consider the estimation of a density at a fixed point under a local differential privacy constraint, where the observations are anonymised before being available for statistical inference. We propose both a privatised version of a projection density estimator as well as a kernel density estimator and derive their minimax rates under a privacy constraint. There is a twofold deterioration of the minimax rates due to the anonymisation, which we show to be unavoidable by providing lower bounds. In both estimation procedures a tuning parameter has to be chosen. We suggest a variant of the classi…
We revisit the classical problem of nonparametric density estimation but impose local differential privacy constraints. Under such constraints, the original multivariate data $X_1,\ldots,X_n \in \mathbb{R}^d$ cannot be directly observed, and all estimators are functions of the randomised output of a suitable privacy mechanism. The statistician is free to choose the form of the privacy mechanism, and in this work we propose to add Laplace distributed noise to a discretisation of the location of a vector $X_i$. Based on these randomised data, we design a novel estimator of the density function, …
We consider the goodness-of-fit testing problem of distinguishing whether the data are drawn from a specified distribution, versus a composite alternative separated from the null in the total variation metric. In the discrete case, we consider goodness-of-fit testing when the null distribution has a possibly growing or unbounded number of categories. In the continuous case, we consider testing a Lipschitz density, with possibly unbounded support, in the low-smoothness regime where the Lipschitz parameter is not assumed to be constant. In contrast to existing results, we show that the minimax r…
We consider the identity testing problem - or goodness-of-fit testing problem - in multivariate binomial families, multivariate Poisson families and multinomial distributions. Given a known distribution $p$ and $n$ iid samples drawn from an unknown distribution $q$, we investigate how large $ρ>0$ should be to distinguish, with high probability, the case $p=q$ from the case $d(p,q) \geq ρ$, where $d$ denotes a specific distance over probability distributions. We answer this question in the case of a family of different distances: $d(p,q) = \|p-q\|_t$ for $t \in [1,2]$ where $\|\cdot\|_t$ is the…
We consider the detection problem of a two-dimensional function from noisy observations of its integrals over lines. We study both rate and sharp asymptotics for the error probabilities in the minimax setup. By construction, the derived tests are non-adaptive. We also construct a minimax rate-optimal adaptive test of rather simple structure.
We observe a sample of $n$ independent $p$-dimensional Gaussian vectors with Toeplitz covariance matrix $ Σ= [σ_{|i-j|}]_{1 \leq i,j \leq p}$ and $σ_0=1$. We consider the problem of testing the hypothesis that $Σ$ is the identity matrix asymptotically when $n \to \infty$ and $p \to \infty$. We suppose that the covariances $σ_k$ decrease either polynomially ($\sum_{k \geq 1} k^{2α} σ^2_{k} \leq L$ for $ α>1/4$ and $L>0$) or exponentially ($\sum_{k \geq 1} e^{2Ak} σ^2_{k} \leq L$ for $ A,L>0$). We consider a test procedure based on a weighted U-statistic of order 2, with optimal weights chosen a…
We study the phase synchronization problem with noisy measurements $Y=z^*z^{*H}+σW\in\mathbb{C}^{n\times n}$, where $z^*$ is an $n$-dimensional complex unit-modulus vector and $W$ is a complex-valued Gaussian random matrix. It is assumed that each entry $Y_{jk}$ is observed with probability $p$. We prove that an SDP relaxation of the MLE achieves the error bound $(1+o(1))\frac{σ^2}{2np}$ under a normalized squared $\ell_2$ loss. This result matches the minimax lower bound of the problem, and even the leading constant is sharp. The analysis of the SDP is based on an equivalent non-convex progra…
Ill-posed inverse problems arise in various scientific fields. We consider the signal detection problem for mildly, severely and extremely ill-posed inverse problems with $l^q$-ellipsoids (bodies), $q\in(0,2]$, for Sobolev, analytic and generalized analytic classes of functions under the Gaussian white noise model. We study both rate and sharp asymptotics for the error probabilities in the minimax setup. By construction, the derived tests are, often, nonadaptive. Minimax rate-optimal adaptive tests of rather simple structure are also constructed.
We study the detection of a sparse change in a high-dimensional mean vector as a minimax testing problem. Our first main contribution is to derive the exact minimax testing rate across all parameter regimes for $n$ independent, $p$-variate Gaussian observations. This rate exhibits a phase transition when the sparsity level is of order $\sqrt{p \log \log (8n)}$ and has a very delicate dependence on the sample size: in a certain sparsity regime it involves a triple iterated logarithmic factor in~$n$. Further, in a dense asymptotic regime, we identify the sharp leading constant, while in the corr…
We consider a compound testing problem within the Gaussian sequence model in which the null and alternative are specified by a pair of closed, convex cones. Such cone testing problem arise in various applications, including detection of treatment effects, trend detection in econometrics, signal detection in radar processing, and shape-constrained inference in non-parametric statistics. We provide a sharp characterization of the GLRT testing radius up to a universal multiplicative constant in terms of the geometric structure of the underlying convex cones. When applied to concrete examples, thi…
The paper is devoted to the problem of estimation of a univariate component in a heteroscedastic nonparametric multiple regression under the mean integrated squared error (MISE) criteria. The aim is to understand how the scale function should be used for estimation of the univariate component. It is known that the scale function does not affect the rate of the MISE convergence, and as a result sharp constants are explored. The paper begins with developing a sharp-minimax theory for a pivotal model $Y=f(X)+σ(X,\mathbf{Z})\varepsilon$, where $\varepsilon$ is standard normal and independent of th…
In one-dimensional density estimation on i.i.d. observations we suggest an adaptive cross-validation technique for the selection of a kernel estimator. This estimator is both asymptotic MISE-efficient with respect to the monotone oracle, and sharp minimax-adaptive over the whole scale of Sobolev spaces with smoothness index greater than 1/2. The proof of the central concentration inequality avoids "chaining" and relies on an additive decomposition of the empirical processes involved.
We consider the problem of comparing probability densities between two groups. A new probabilistic tensor product smoothing spline framework is developed to model the joint density of two variables. Under such a framework, the probability density comparison is equivalent to testing the presence/absence of interactions. We propose a penalized likelihood ratio test for such interaction testing and show that the test statistic is asymptotically chi-square distributed under the null hypothesis. Furthermore, we derive a sharp minimax testing rate based on the Bernstein width for nonparametric two-s…
We consider the problem of testing a particular type of composite null hypothesis under a nonparametric multivariate regression model. For a given quadratic functional $Q$, the null hypothesis states that the regression function $f$ satisfies the constraint $Q[f]=0$, while the alternative corresponds to the functions for which $Q[f]$ is bounded away from zero. On the one hand, we provide minimax rates of testing and the exact separation constants, along with a sharp-optimal testing procedure, for diagonal and nonnegative quadratic functionals. We consider smoothness classes of ellipsoidal form…
We study sparse principal components analysis in the high-dimensional setting, where $p$ (the number of variables) can be much larger than $n$ (the number of observations). We prove optimal, non-asymptotic lower and upper bounds on the minimax estimation error for the leading eigenvector when it belongs to an $\ell_q$ ball for $q \in [0,1]$. Our bounds are sharp in $p$ and $n$ for all $q \in [0, 1]$ over a wide class of distributions. The upper bound is obtained by analyzing the performance of $\ell_q$-constrained PCA. In particular, our results provide convergence rates for $\ell_1$-constrain…
We consider testing for presence of a signal in Gaussian white noise with intensity 1/sqrt(n), when the alternatives are given by smoothness ellipsoids with an L2-ball of (squared) radius rho removed. It is known that, for a fixed Sobolev type ellipsoid of smoothness beta and size M, a rho which is of order n to the power -4 beta/(4 beta+1)} is the critical separation rate, in the sense that the minimax error of second kind over alpha-tests stays asymptotically between 0 and 1 strictly (Ingster, 1982). In addition, Ermakov (1990) found the sharp asymptotics of the minimax error of second kind …
We address the problem of non-parametric density estimation under the additional constraint that only privatised data are allowed to be published and available for inference. For this purpose, we adopt a recent generalisation of classical minimax theory to the framework of local $α$-differential privacy and provide a lower bound on the rate of convergence over Besov spaces $B^s_{pq}$ under mean integrated $\mathbb L^r$-risk. This lower bound is deteriorated compared to the standard setup without privacy, and reveals a twofold elbow effect. In order to fulfil the privacy requirement, we suggest…
We consider a matrix-valued Gaussian sequence model, that is, we observe a sequence of high-dimensional $M \times N$ matrices of heterogeneous Gaussian random variables $x_{ij,k}$ for $i \in\{1,...,M\}$, $j \in \{1,...,N\}$ and $k \in \mathbb{Z}$. The standard deviation of our observations is $\ep k^s$ for some $\ep >0$ and $s \geq 0$. We give sharp rates for the detection of a sparse submatrix of size $m \times n$ with active components. A component $(i,j)$ is said active if the sequence $\{x_{ij,k}\}_k$ have mean $\{θ_{ij,k}\}_k$ within a Sobolev ellipsoid of smoothness $τ>0$ and total energ…
The question of fast convergence in the classical problem of high dimensional linear regression has been extensively studied. Arguably, one of the fastest procedures in practice is Iterative Hard Thresholding (IHT). Still, IHT relies strongly on the knowledge of the true sparsity parameter $s$. In this paper, we present a novel fast procedure for estimation in the high dimensional linear regression. Taking advantage of the interplay between estimation, support recovery and optimization we achieve both optimal statistical accuracy and fast convergence. The main advantage of our procedure is tha…
In a sparse high-dimensional elliptical model we consider a hard threshold estimator for the correlation matrix based on Kendall's tau with threshold level $α(\frac{\log p}{n})^{1/2}$. Parameters $α$ are identified such that the threshold estimator achieves the minimax rate under the squared Frobenius norm and the squared spectral norm. This allows a reasonable calibration of the estimator without any quantitative information about the tails of the underlying distribution. For Gaussian observations we even establish a critical threshold constant $α^\ast$ under the squared Frobenius norm, i.e. …
We consider random-design linear prediction and related questions on the lower tail of random matrices. It is known that, under boundedness constraints, the minimax risk is of order $d/n$ in dimension $d$ with $n$ samples. Here, we study the minimax expected excess risk over the full linear class, depending on the distribution of covariates. First, the least squares estimator is exactly minimax optimal in the well-specified case, for every distribution of covariates. We express the minimax risk in terms of the distribution of statistical leverage scores of individual samples, and deduce a mini…
University of Pennsylvania（宾夕法尼亚大学）；Cornell University（康奈尔大学）；CREST-ENSAE（法国国家统计与经济管理学院（CREST-ENSAE））
We consider the random design regression model with square loss. We propose a method that aggregates empirical minimizers (ERM) over appropriately chosen random subsets and reduces to ERM in the extreme case, and we establish sharp oracle inequalities for its risk. We show that, under the $\varepsilon^{-p}$ growth of the empirical $\varepsilon$-entropy, the excess risk of the proposed method attains the rate $n^{-2/(2+p)}$ for $p\in(0,2)$ and $n^{-1/p}$ for $p>2$ where $n$ is the sample size. Furthermore, for $p\in(0,2)$, the excess risk rate matches the behavior of the minimax risk of functio…
Since Stein's 1956 seminal paper, shrinkage has played a fundamental role in both parametric and nonparametric inference. This article discusses minimaxity and adaptive minimaxity in nonparametric function estimation. Three interrelated problems, function estimation under global integrated squared error, estimation under pointwise squared error, and nonparametric confidence intervals, are considered. Shrinkage is pivotal in the development of both the minimax theory and the adaptation theory. While the three problems are closely connected and the minimax theories bear some similarities, the ad…
Harvard University（哈佛大学）；Instituto de Investigaciones en Matemáticas Aplicadas y en Sistemas (IIMAS), Universidad Nacional Autónoma de México（墨西哥国立自治大学应用数学与系统研究所（IIMAS））
We investigate the contraction coefficients derived from strong data processing inequalities for the $E_γ$-divergence. By generalizing the celebrated Dobrushin's coefficient from total variation distance to $E_γ$-divergence, we derive a closed-form expression for the contraction of $E_γ$-divergence. This result has fundamental consequences in two privacy settings. First, it implies that local differential privacy can be equivalently expressed in terms of the contraction of $E_γ$-divergence. This equivalent formula can be used to precisely quantify the impact of local privacy in (Bayesian and m…
