Standard approaches to stochastic gradient estimation, with only noisy black-box function evaluations, use the finite-difference method or its variants. While natural, it is open to our knowledge whether their statistical accuracy is the best possible. This paper argues so by showing that central finite-difference is a nearly minimax optimal zeroth-order gradient estimator for a suitable class of objective functions and mean squared risk, among both the class of linear estimators and the much larger class of all (nonlinear) estimators.
In this paper, we are interested in the study of beta kernel estimators from an asymptotic minimax point of view. It is well known that beta kernel estimators are, on the contrary of classical kernel estimators, "free of boundary effect" and thus are very useful in practice. The goal of this paper is to prove that there is a price to pay: for very regular functions or for certain losses, these estimators are not minimax. Nevertheless they are minimax for classical regularities such as regularity of order two or less than two, supposed commonly in the practice and for some classical losses.
In this paper, we observe a sparse mean vector through Gaussian noise and we aim at estimating some additive functional of the mean in the minimax sense. More precisely, we generalize the results of (Collier et al., 2017, 2019) to a very large class of functionals. The optimal minimax rate is shown to depend on the polynomial approximation rate of the marginal functional, and optimal estimators achieving this rate are built.
In this paper, we introduce a new estimator for the emission densities of a nonparametric hidden Markov model. It is adaptive and minimax with respect to each state's regularity--as opposed to globally minimax estimators, which adapt to the worst regularity among the emission densities. Our method is based on Goldenshluger and Lepski's methodology. It is computationally efficient and only requires a family of preliminary estimators, without any restriction on the type of estimators considered. We present two such estimators that allow to reach minimax rates up to a logarithmic term: a spectral…
For the problem of nonparametric estimation of signal in Gaussian noise we point out the strong asymptotically minimax estimators on maxisets for linear estimators (see \cite{ker93,rio}). It turns out that the order of rates of convergence of Pinsker estimator on this maxisets is worse than the order of rates of convergence for the class of linear estimators considered on this maxisets. We show that balls in Sobolev spaces are maxisets for Pinsker estimators.
In this article, we consider the problem of inverting the exponential Radon transform of a function in the presence of noise. We propose a kernel estimator to estimate the true function, analogous to the one proposed by Korostelëv and Tsybakov in their article `Optimal rates of convergence of estimators in a probabilistic setup of tomography problem', Problems of Information Transmission, 27:73-81,1991. For the estimator proposed in this article, we then show that it converges to the true function at a minimax optimal rate.
This paper considers the asymptotic behavior in $β$-Hölder spaces, and under $L^p$ losses, of a Dirichlet kernel density estimator proposed by Aitchison and Lauder (1985) for the analysis of compositional data. In recent work, Ouimet and Tolosana-Delgado (2022) established the uniform strong consistency and asymptotic normality of this estimator. As a complement, it is shown here that the Aitchison-Lauder estimator can achieve the minimax rate asymptotically for a suitable choice of bandwidth whenever $(p,β) \in [1, 3) \times (0, 2]$ or $(p, β) \in \mathcal{A}_d$, where $\mathcal{A}_d$ is a sp…
We address the problem of adaptive minimax density estimation on $\bR^d$ with $\bL_p$--loss on the anisotropic Nikol'skii classes. We fully characterize behavior of the minimax risk for different relationships between regularity parameters and norm indexes in definitions of the functional class and of the risk. In particular, we show that there are four different regimes with respect to the behavior of the minimax risk. We develop a single estimator which is (nearly) optimal in orderover the complete scale of the anisotropic Nikol'skii classes. Our estimation procedure is based on a data-drive…
We present estimators for a well studied statistical estimation problem: the estimation for the linear regression model with soft sparsity constraints ($\ell_q$ constraint with $0<q\leq1$) in the high-dimensional setting. We first present a family of estimators, called the projected nearest neighbor estimator and show, by using results from Convex Geometry, that such estimator is within a logarithmic factor of the optimal for any design matrix. Then by utilizing a semi-definite programming relaxation technique developed in [SIAM J. Comput. 36 (2007) 1764-1776], we obtain an approximation algor…
Missing data occur frequently in a wide range of applications. In this paper, we consider estimation of high-dimensional covariance matrices in the presence of missing observations under a general missing completely at random model in the sense that the missingness is not dependent on the values of the data. Based on incomplete data, estimators for bandable and sparse covariance matrices are proposed and their theoretical and numerical properties are investigated. Minimax rates of convergence are established under the spectral norm loss and the proposed estimators are shown to be rate-optimal …
In this paper, we consider estimators for an additive functional of $ϕ$, which is defined as $θ(P;ϕ)=\sum_{i=1}^kϕ(p_i)$, from $n$ i.i.d. random samples drawn from a discrete distribution $P=(p_1,...,p_k)$ with alphabet size $k$. We propose a minimax optimal estimator for the estimation problem of the additive functional. We reveal that the minimax optimal rate is characterized by the divergence speed of the fourth derivative of $ϕ$ if the divergence speed is high. As a result, we show there is no consistent estimator if the divergence speed of the fourth derivative of $ϕ$ is larger than $p^{-…
We revisit the classical problem of nonparametric density estimation but impose local differential privacy constraints. Under such constraints, the original multivariate data $X_1,\ldots,X_n \in \mathbb{R}^d$ cannot be directly observed, and all estimators are functions of the randomised output of a suitable privacy mechanism. The statistician is free to choose the form of the privacy mechanism, and in this work we propose to add Laplace distributed noise to a discretisation of the location of a vector $X_i$. Based on these randomised data, we design a novel estimator of the density function, …
We consider the problem of estimating the regression function in functional linear regression models by proposing a new type of projection estimators which combine dimension reduction and thresholding. The introduction of a threshold rule allows to get consistency under broad assumptions as well as minimax rates of convergence under additional regularity hypotheses. We also consider the particular case of Sobolev spaces generated by the trigonometric basis which permits to get easily mean squared error of prediction as well as estimators of the derivatives of the regression function. We prove …
We consider the problem of designing minimax estimators for estimating the parameters of a probability distribution. Unlike classical approaches such as the MLE and minimum distance estimators, we consider an algorithmic approach for constructing such estimators. We view the problem of designing minimax estimators as finding a mixed strategy Nash equilibrium of a zero-sum game. By leveraging recent results in online learning with non-convex losses, we provide a general algorithm for finding a mixed-strategy Nash equilibrium of general non-convex non-concave zero-sum games. Our algorithm requir…
This article studies nonparametric methods to estimate the co-integrated volatility for multi-dimensional Lévy processes with high frequency data. We construct a spectral estimator for the co-integrated volatility and prove minimax rates for an appropriate bounded nonparametric class of semimartingales. Given $ n $ observations of increments over intervals of length $1/n$, the rates of convergence are $1 / \sqrt{n} $ if $ r \leq 1 $ and $ (n\log n)^{(r-2)/2} $ if $ r>1 $, which are optimal in a minimax sense. We bound the co-jump index activity from below with the harmonic mean. Finally, we as…
We describe the minimax reconstruction rates in linear ill-posed equations in Hilbert space when smoothness is given in terms of general source sets. The underlying fundamental result, the minimax rate on ellipsoids, is proved similarly to the seminal study by D. L. Donoho, R.~C. Liu, and B. MacGibbon, {\it Minimax risk over hyperrectangles, and implications}, Ann.~ Statist. 18, 1990. These authors highlighted the special role of the truncated series estimator, and for such estimators the risk can explicitly be given. We provide several examples, indicating results for statistical estimation i…
An Efficient Minimax Optimal Estimator For Multivariate Convex Regression
Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）；Department of Mathematics, Tel Aviv University（特拉维夫大学数学系）
We address the problem of adaptive minimax estimation in white gaussian noise model under $L_p$--loss, $1\leq p\leq\infty,$ on the anisotropic Nikolskii classes. We present the estimation procedure based on a new data-driven selection scheme from the family of kernel estimators with varying bandwidths. For proposed estimator we establish so-called Lp-norm oracle inequality and use it for deriving minimax adaptive results. We prove the existence of rate-adaptive estimators and fully characterize behavior of the minimax risk for different relationships between regularity parameters and norm inde…
This paper studies the minimax rate of nonparametric conditional density estimation under a weighted absolute value loss function in a multivariate setting. We first demonstrate that conditional density estimation is impossible if one only requires that $p_{X|Z}$ is smooth in $x$ for all values of $z$. This motivates us to consider a sub-class of absolutely continuous distributions, restricting the conditional density $p_{X|Z}(x|z)$ to not only be Hölder smooth in $x$, but also be total variation smooth in $z$. We propose a corresponding kernel-based estimator and prove that it achieves the mi…
In this paper, decision theory was used to derive Bayes and minimax decision rules to estimate allelic frequencies and to explore their admissibility. Decision rules with uniformly smallest risk usually do not exist and one approach to solve this problem is to use the Bayes principle and the minimax principle to find decision rules satisfying some general optimality criterion based on their risk functions. Two cases were considered, the simpler case of biallelic loci and the more complex case of multiallelic loci. For each locus, the sampling model was a multinomial distribution and the prior …
This paper characterizes the minimax linear estimator of the value of an unknown function at a boundary point of its domain in a Gaussian white noise model under the restriction that the first-order derivative of the unknown function is Lipschitz continuous (the second-order Hölder class). The result is then applied to construct the minimax optimal estimator for the regression discontinuity design model, where the parameter of interest involves function values at boundary points.
Inference about dependencies in a multiway data array can be made using the array normal model, which corresponds to the class of multivariate normal distributions with separable covariance matrices. Maximum likelihood and Bayesian methods for inference in the array normal model have appeared in the literature, but there have not been any results concerning the optimality properties of such estimators. In this article, we obtain results for the array normal model that are analogous to some classical results concerning covariance estimation for the multivariate normal model. We show that under …
We study the problem of aggregation of estimators when the estimators are not independent of the data used for aggregation and no sample splitting is allowed. If the estimators are deterministic vectors, it is well known that the minimax rate of aggregation is of order $\log(M)$, where $M$ is the number of estimators to aggregate. It is proved that for affine estimators, the minimax rate of aggregation is unchanged: it is possible to handle the linear dependence between the affine estimators and the data used for aggregation at no extra cost. The minimax rate is not impacted either by the vari…
We consider the problem of estimating the predictive density of future observations from a non-parametric regression model. The density estimators are evaluated under Kullback--Leibler divergence and our focus is on establishing the exact asymptotics of minimax risk in the case of Gaussian errors. We derive the convergence rate and constant for minimax risk among Bayesian predictive densities under Gaussian priors and we show that this minimax risk is asymptotically equivalent to that among all density estimators.
In a sparse high-dimensional elliptical model we consider a hard threshold estimator for the correlation matrix based on Kendall's tau with threshold level $α(\frac{\log p}{n})^{1/2}$. Parameters $α$ are identified such that the threshold estimator achieves the minimax rate under the squared Frobenius norm and the squared spectral norm. This allows a reasonable calibration of the estimator without any quantitative information about the tails of the underlying distribution. For Gaussian observations we even establish a critical threshold constant $α^\ast$ under the squared Frobenius norm, i.e. …
We analyze the Kozachenko--Leonenko (KL) nearest neighbor estimator for the differential entropy. We obtain the first uniform upper bound on its performance over Hölder balls on a torus without assuming any conditions on how close the density could be from zero. Accompanying a new minimax lower bound over the Hölder ball, we show that the KL estimator is achieving the minimax rates up to logarithmic factors without cognizance of the smoothness parameter $s$ of the Hölder ball for $s\in (0,2]$ and arbitrary dimension $d$, rendering it the first estimator that provably satisfies this property.
We study the convergence properties, in Hellinger and related distances, of nonparametric density estimators based on measure transport. These estimators represent the measure of interest as the pushforward of a chosen reference distribution under a transport map, where the map is chosen via a maximum likelihood objective (equivalently, minimizing an empirical Kullback-Leibler loss) or a penalized version thereof. We establish concentration inequalities for a general class of penalized measure transport estimators, by combining techniques from M-estimation with analytical properties of the tra…
Crowdsourcing has become a primary means for label collection in many real-world machine learning applications. A classical method for inferring the true labels from the noisy labels provided by crowdsourcing workers is Dawid-Skene estimator. In this paper, we prove convergence rates of a projected EM algorithm for the Dawid-Skene estimator. The revealed exponent in the rate of convergence is shown to be optimal via a lower bound argument. Our work resolves the long standing issue of whether Dawid-Skene estimator has sound theoretical guarantees besides its good performance observed in practic…
CEREMADE（CEREMADE（巴黎多芬大学-PSL大学））；Université Paris Dauphine - PSL（巴黎多芬大学-PSL大学）；CREST；ENSAE（法国国家统计与经济管理学院）；IP Paris（巴黎综合理工学院联盟）
We study the problem of robust estimation of the mean vector of a sub-Gaussian distribution. We introduce an estimator based on spectral dimension reduction (SDR) and establish a finite sample upper bound on its error that is minimax-optimal up to a logarithmic factor. Furthermore, we prove that the breakdown point of the SDR estimator is equal to $1/2$, the highest possible value of the breakdown point. In addition, the SDR estimator is equivariant by similarity transforms and has low computational complexity. More precisely, in the case of $n$ vectors of dimension $p$ -- at most $\varepsilon…
We study the problem of estimating a mean pattern from a set of similar curves in the setting where the variability in the data is due to random geometric deformations and additive noise. We propose an estimator based on the notion of Frechet mean that is a generalization of the standard notion of averaging to non-Euclidean spaces. We derive a minimax rate for this estimation problem, and we show that our estimator achieves this optimal rate under the asymptotics where both the number of curves and the number of sampling points go to infinity.
We consider high-dimensional multivariate linear regression models, where the joint distribution of covariates and response variables is a multivariate normal distribution with a bandable covariance matrix. The main goal of this paper is to estimate the regression coefficient matrix, which is a function of the bandable covariance matrix. Although the tapering estimator of covariance has the minimax optimal convergence rate for the class of bandable covariances, we show that it has a sub-optimal convergence rate for the regression coefficient; that is, a minimax estimator for the class of banda…
In this work we study the semi-supervised framework of confidence set classification with controlled expected size in minimax settings. We obtain semi-supervised minimax rates of convergence under the margin assumption and a H{ö}lder condition on the regression function. Besides, we show that if no further assumptions are made, there is no supervised method that outperforms the semi-supervised estimator proposed in this work. We establish that the best achievable rate for any supervised method is n^{--1/2} , even if the margin assumption is extremely favorable. On the contrary, semi-supervised…
In this paper we study minimax and adaptation rates in general isotonic regression. For uniform deterministic and random designs in $[0,1]^d$ with $d\ge 2$ and $N(0,1)$ noise, the minimax rate for the $\ell_2$ risk is known to be bounded from below by $n^{-1/d}$ when the unknown mean function $f$ is nondecreasing and its range is bounded by a constant, while the least squares estimator (LSE) is known to nearly achieve the minimax rate up to a factor $(\log n)^γ$ where $n$ is sample size, $γ= 4$ in the lattice design and $γ= \max\{9/2, (d^2+d+1)/2 \}$ in the random design. Moreover, the LSE is …
The IEEE 1588 protocol has received recent interest as a means of delivering sub-microsecond level clock phase synchronization over packet-switched mobile backhaul networks. Due to the randomness of the end-to-end delays in packet networks, the recovery of clock phase from packet timestamps in IEEE 1588 must be treated as a statistical estimation problem. A number of estimators for this problem have been suggested in the literature, but little is known about the best achievable performance. In this paper, we describe new minimax estimators for this problem, that are optimum in terms of minimiz…
We study minimax rates for high-dimensional linear regression with additive errors under the $\ell_p\ (1\leq p<\infty)$-losses, where the regression parameter is of weak sparsity. Our lower and upper bounds agree up to constant factors, implying that the proposed estimator is minimax optimal.
We study the minimax estimation of $α$-divergences between discrete distributions for integer $α\ge 1$, which include the Kullback--Leibler divergence and the $χ^2$-divergences as special examples. Dropping the usual theoretical tricks to acquire independence, we construct the first minimax rate-optimal estimator which does not require any Poissonization, sample splitting, or explicit construction of approximating polynomials. The estimator uses a hybrid approach which solves a problem-independent linear program based on moment matching in the non-smooth regime, and applies a problem-dependent…
We consider the problem of estimating the $L_1$ distance between two discrete probability measures $P$ and $Q$ from empirical data in a nonasymptotic and large alphabet setting. When $Q$ is known and one obtains $n$ samples from $P$, we show that for every $Q$, the minimax rate-optimal estimator with $n$ samples achieves performance comparable to that of the maximum likelihood estimator (MLE) with $n\ln n$ samples. When both $P$ and $Q$ are unknown, we construct minimax rate-optimal estimators whose worst case performance is essentially that of the known $Q$ case with $Q$ being uniform, implyi…
In this contribution we introduce weakly locally stationary time series through the local approximation of the non-stationary covariance structure by a stationary one. This allows us to define autoregression coefficients in a non-stationary context, which, in the particular case of a locally stationary Time Varying Autoregressive (TVAR) process, coincide with the generating coefficients. We provide and study an estimator of the time varying autoregression coefficients in a general setting. The proposed estimator of these coefficients enjoys an optimal minimax convergence rate under limited smo…
We address the problem of density estimation with $\mathbb{L}_s$-loss by selection of kernel estimators. We develop a selection procedure and derive corresponding $\mathbb{L}_s$-risk oracle inequalities. It is shown that the proposed selection rule leads to the estimator being minimax adaptive over a scale of the anisotropic Nikol'skii classes. The main technical tools used in our derivations are uniform bounds on the $\mathbb{L}_s$-norms of empirical processes developed recently by Goldenshluger and Lepski [Ann. Probab. (2011), to appear].
We consider the problem of estimating a function defined over $n$ locations on a $d$-dimensional grid (having all side lengths equal to $n^{1/d}$). When the function is constrained to have discrete total variation bounded by $C_n$, we derive the minimax optimal (squared) $\ell_2$ estimation error rate, parametrized by $n$ and $C_n$. Total variation denoising, also known as the fused lasso, is seen to be rate optimal. Several simpler estimators exist, such as Laplacian smoothing and Laplacian eigenmaps. A natural question is: can these simpler estimators perform just as well? We prove that thes…
We present new estimators of the mean of a real valued random variable, based on PAC-Bayesian iterative truncation. We analyze the non-asymptotic minimax properties of the deviations of estimators for distributions having either a bounded variance or a bounded kurtosis. It turns out that these minimax deviations are of the same order as the deviations of the empirical mean estimator of a Gaussian distribution. Nevertheless, the empirical mean itself performs poorly at high confidence levels for the worst distribution with a given variance or kurtosis (which turns out to be heavy tailed). To ob…
We propose a general methodology for the construction and analysis of minimax estimators for a wide class of functionals of finite dimensional parameters, and elaborate on the case of discrete distributions, where the alphabet size $S$ is unknown and may be comparable with the number of observations $n$. We treat the respective regions where the functional is "nonsmooth" and "smooth" separately. In the "nonsmooth" regime, we apply an unbiased estimator for the best polynomial approximation of the functional whereas, in the "smooth" regime, we apply a bias-corrected Maximum Likelihood Estimator…
We estimate convex polytopes and general convex sets in $\mathbb R^d,d\geq 2$ in the regression framework. We measure the risk of our estimators using a $L^1$-type loss function and prove upper bounds on these risks. We show that, in the case of polytopes, these estimators achieve the minimax rate. For polytopes, this minimax rate is $\frac{\ln n}{n}$, which differs from the parametric rate for non-regular families by a logarithmic factor, and we show that this extra factor is essential. Using polytopal approximations we extend our results to general convex sets, and we achieve the minimax rat…
Consider the problem of estimating the Shannon entropy of a distribution over $k$ elements from $n$ independent samples. We show that the minimax mean-square error is within universal multiplicative constant factors of $$\Big(\frac{k }{n \log k}\Big)^2 + \frac{\log^2 k}{n}$$ if $n$ exceeds a constant factor of $\frac{k}{\log k}$; otherwise there exists no consistent estimator. This refines the recent result of Valiant-Valiant \cite{VV11} that the minimal sample size for consistent entropy estimation scales according to $Θ(\frac{k}{\log k})$. The apparatus of best polynomial approximation plays…
The purpose of this paper is to study the problem of estimating a compactly supported density of probability from noisy observations of its moments. In fact, we provide a statistical approach to the famous Hausdorff classical moment problem. We prove an upper bound and a lower bound on the rate of convergence of the mean squared error showing that the considered estimator attains minimax rate over the corresponding smoothness classes.
