An Efficient Minimax Optimal Estimator For Multivariate Convex Regression
Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）；Department of Mathematics, Tel Aviv University（特拉维夫大学数学系）
Gil Kur, Eli Putterman
备注：
Minor corrections and improved presentation (appeared at COLT 2022)
This work studies the computational aspects of multivariate convex regression in dimensions $d \ge 5$. Our results include the \emph{first} estimators that are minimax optimal (up to logarithmic factors) with polynomial runtime in the sample size for both $L$-Lipschitz convex regression, and $Γ$-bounded convex regression under polytopal support. Our analysis combines techniques from empirical process theory, stochastic geometry, and potential theory, and leverages recent algorithmic advances in mean estimation for random vectors and in distribution-free linear regression. These results provide the first efficient, minimax-optimal procedures for non-Donsker classes for which their corresponding least-squares estimator is provably minimax-suboptimal.
We prove minimax bounds for estimating Gaussian location mixtures on $\mathbb{R}^d$ under the squared $L^2$ and the squared Hellinger loss functions. Under the squared $L^2$ loss, we prove that the minimax rate is upper and lower bounded by a constant multiple of $n^{-1}(\log n)^{d/2}$. Under the squared Hellinger loss, we consider two subclasses based on the behavior of the tails of the mixing measure. When the mixing measure has a sub-Gaussian tail, the minimax rate under the squared Hellinger loss is bounded from below by $(\log n)^{d}/n$. On the other hand, when the mixing measure is only …
We consider the identity testing problem - or goodness-of-fit testing problem - in multivariate binomial families, multivariate Poisson families and multinomial distributions. Given a known distribution $p$ and $n$ iid samples drawn from an unknown distribution $q$, we investigate how large $ρ>0$ should be to distinguish, with high probability, the case $p=q$ from the case $d(p,q) \geq ρ$, where $d$ denotes a specific distance over probability distributions. We answer this question in the case of a family of different distances: $d(p,q) = \|p-q\|_t$ for $t \in [1,2]$ where $\|\cdot\|_t$ is the…
In this paper, we observe a sparse mean vector through Gaussian noise and we aim at estimating some additive functional of the mean in the minimax sense. More precisely, we generalize the results of (Collier et al., 2017, 2019) to a very large class of functionals. The optimal minimax rate is shown to depend on the polynomial approximation rate of the marginal functional, and optimal estimators achieving this rate are built.
In this paper, we introduce a new estimator for the emission densities of a nonparametric hidden Markov model. It is adaptive and minimax with respect to each state's regularity--as opposed to globally minimax estimators, which adapt to the worst regularity among the emission densities. Our method is based on Goldenshluger and Lepski's methodology. It is computationally efficient and only requires a family of preliminary estimators, without any restriction on the type of estimators considered. We present two such estimators that allow to reach minimax rates up to a logarithmic term: a spectral…
In this work, we propose an efficient minimax optimal global optimization algorithm for multivariate Lipschitz continuous functions. To evaluate the performance of our approach, we utilize the average regret instead of the traditional simple regret, which, as we show, is not suitable for use in the multivariate non-convex optimization because of the inherent hardness of the problem itself. Since we study the average regret of the algorithm, our results directly imply a bound for the simple regret as well. Instead of constructing lower bounding proxy functions, our method utilizes a predetermin…
We revisit the classical problem of nonparametric density estimation but impose local differential privacy constraints. Under such constraints, the original multivariate data $X_1,\ldots,X_n \in \mathbb{R}^d$ cannot be directly observed, and all estimators are functions of the randomised output of a suitable privacy mechanism. The statistician is free to choose the form of the privacy mechanism, and in this work we propose to add Laplace distributed noise to a discretisation of the location of a vector $X_i$. Based on these randomised data, we design a novel estimator of the density function, …
In this article, we consider the problem of inverting the exponential Radon transform of a function in the presence of noise. We propose a kernel estimator to estimate the true function, analogous to the one proposed by Korostelëv and Tsybakov in their article `Optimal rates of convergence of estimators in a probabilistic setup of tomography problem', Problems of Information Transmission, 27:73-81,1991. For the estimator proposed in this article, we then show that it converges to the true function at a minimax optimal rate.
Standard approaches to stochastic gradient estimation, with only noisy black-box function evaluations, use the finite-difference method or its variants. While natural, it is open to our knowledge whether their statistical accuracy is the best possible. This paper argues so by showing that central finite-difference is a nearly minimax optimal zeroth-order gradient estimator for a suitable class of objective functions and mean squared risk, among both the class of linear estimators and the much larger class of all (nonlinear) estimators.
This paper considers the asymptotic behavior in $β$-Hölder spaces, and under $L^p$ losses, of a Dirichlet kernel density estimator proposed by Aitchison and Lauder (1985) for the analysis of compositional data. In recent work, Ouimet and Tolosana-Delgado (2022) established the uniform strong consistency and asymptotic normality of this estimator. As a complement, it is shown here that the Aitchison-Lauder estimator can achieve the minimax rate asymptotically for a suitable choice of bandwidth whenever $(p,β) \in [1, 3) \times (0, 2]$ or $(p, β) \in \mathcal{A}_d$, where $\mathcal{A}_d$ is a sp…
In this paper, we consider estimators for an additive functional of $ϕ$, which is defined as $θ(P;ϕ)=\sum_{i=1}^kϕ(p_i)$, from $n$ i.i.d. random samples drawn from a discrete distribution $P=(p_1,...,p_k)$ with alphabet size $k$. We propose a minimax optimal estimator for the estimation problem of the additive functional. We reveal that the minimax optimal rate is characterized by the divergence speed of the fourth derivative of $ϕ$ if the divergence speed is high. As a result, we show there is no consistent estimator if the divergence speed of the fourth derivative of $ϕ$ is larger than $p^{-…
We consider the problem of designing minimax estimators for estimating the parameters of a probability distribution. Unlike classical approaches such as the MLE and minimum distance estimators, we consider an algorithmic approach for constructing such estimators. We view the problem of designing minimax estimators as finding a mixed strategy Nash equilibrium of a zero-sum game. By leveraging recent results in online learning with non-convex losses, we provide a general algorithm for finding a mixed-strategy Nash equilibrium of general non-convex non-concave zero-sum games. Our algorithm requir…
CEREMADE（CEREMADE（巴黎多芬大学-PSL大学））；Université Paris Dauphine - PSL（巴黎多芬大学-PSL大学）；CREST；ENSAE（法国国家统计与经济管理学院）；IP Paris（巴黎综合理工学院联盟）
We study the problem of robust estimation of the mean vector of a sub-Gaussian distribution. We introduce an estimator based on spectral dimension reduction (SDR) and establish a finite sample upper bound on its error that is minimax-optimal up to a logarithmic factor. Furthermore, we prove that the breakdown point of the SDR estimator is equal to $1/2$, the highest possible value of the breakdown point. In addition, the SDR estimator is equivariant by similarity transforms and has low computational complexity. More precisely, in the case of $n$ vectors of dimension $p$ -- at most $\varepsilon…
Nearly optimal minimax estimator for high-dimensional sparse linear regression
我们提出了一个很好的研究统计估计问题的估计：估计的线性回归模型与软稀疏约束（$\ell_q$约束与$0<q\leq1$）在高维设置。我们首先提出了一个家庭的估计，称为投影最近邻估计，并显示，通过使用凸几何的结果，这种估计是在一个对数因子的最佳任何设计矩阵。然后利用[SIAM J. COMPUT. 36（2007）1764-1776]，我们获得了用于计算任何这样的估计任务的极大极小风险的近似算法，并且还获得了用于$\ell_1$稀疏约束的重要情况的多项式时间近似最优估计器。尽管对这个问题进行了数十年的研究，但以前只知道特殊情况下的结果。当参数半径未知时，我们也将该方法推广到自适应的情况。
We analyze the Kozachenko--Leonenko (KL) nearest neighbor estimator for the differential entropy. We obtain the first uniform upper bound on its performance over Hölder balls on a torus without assuming any conditions on how close the density could be from zero. Accompanying a new minimax lower bound over the Hölder ball, we show that the KL estimator is achieving the minimax rates up to logarithmic factors without cognizance of the smoothness parameter $s$ of the Hölder ball for $s\in (0,2]$ and arbitrary dimension $d$, rendering it the first estimator that provably satisfies this property.
This article studies nonparametric methods to estimate the co-integrated volatility for multi-dimensional Lévy processes with high frequency data. We construct a spectral estimator for the co-integrated volatility and prove minimax rates for an appropriate bounded nonparametric class of semimartingales. Given $ n $ observations of increments over intervals of length $1/n$, the rates of convergence are $1 / \sqrt{n} $ if $ r \leq 1 $ and $ (n\log n)^{(r-2)/2} $ if $ r>1 $, which are optimal in a minimax sense. We bound the co-jump index activity from below with the harmonic mean. Finally, we as…
Minimax rates for statistical inverse problems under general source conditions
当光滑性由一般源集给出时，我们描述了希尔伯特空间中线性不适定方程组的极小极大重构率。基本结果，椭球上的极大极小率，类似于D。L.多诺霍河~ C. Liu和B. MacGiant，{ \it Minimax risk over hyperrectanges，and implications}，Ann.~中央集权主义者1990年18日这些作者强调了截断序列估计量的特殊作用，对于这种估计量，可以明确地给出风险。我们提供了几个例子，表明结果统计估计不适定的问题在希尔伯特空间。
We study the problem of aggregation of estimators when the estimators are not independent of the data used for aggregation and no sample splitting is allowed. If the estimators are deterministic vectors, it is well known that the minimax rate of aggregation is of order $\log(M)$, where $M$ is the number of estimators to aggregate. It is proved that for affine estimators, the minimax rate of aggregation is unchanged: it is possible to handle the linear dependence between the affine estimators and the data used for aggregation at no extra cost. The minimax rate is not impacted either by the vari…
We study the convergence properties, in Hellinger and related distances, of nonparametric density estimators based on measure transport. These estimators represent the measure of interest as the pushforward of a chosen reference distribution under a transport map, where the map is chosen via a maximum likelihood objective (equivalently, minimizing an empirical Kullback-Leibler loss) or a penalized version thereof. We establish concentration inequalities for a general class of penalized measure transport estimators, by combining techniques from M-estimation with analytical properties of the tra…
This paper studies the minimax rate of nonparametric conditional density estimation under a weighted absolute value loss function in a multivariate setting. We first demonstrate that conditional density estimation is impossible if one only requires that $p_{X|Z}$ is smooth in $x$ for all values of $z$. This motivates us to consider a sub-class of absolutely continuous distributions, restricting the conditional density $p_{X|Z}(x|z)$ to not only be Hölder smooth in $x$, but also be total variation smooth in $z$. We propose a corresponding kernel-based estimator and prove that it achieves the mi…
In this work, the estimation of the multivariate normal mean by different classes of shrinkage estimators is investigated. The risk associated with the balanced loss function is used to compare two estimators. We start by considering estimators that generalize the James-Stein estimator and show that these estimators dominate the maximum likelihood estimator (MLE), therefore are minimax, when the shrinkage function satisfies some conditions. Then, we treat estimators of polynomial form and prove the increase of the degree of the polynomial allows us to build a better estimator from the one prev…
In this paper, the maximum spacing method is considered for multivariate observations. Nearest neighbour balls are used as a multidimensional analogue to univariate spacings. A class of information-type measures is used to generalize the concept of maximum spacing estimators. Asymptotic normality of these generalized maximum spacing estimators is proved when the assigned model class is correct, that is the true density is a member of the model class.
We study the minimax estimation of $α$-divergences between discrete distributions for integer $α\ge 1$, which include the Kullback--Leibler divergence and the $χ^2$-divergences as special examples. Dropping the usual theoretical tricks to acquire independence, we construct the first minimax rate-optimal estimator which does not require any Poissonization, sample splitting, or explicit construction of approximating polynomials. The estimator uses a hybrid approach which solves a problem-independent linear program based on moment matching in the non-smooth regime, and applies a problem-dependent…
We consider the problem of estimating the $L_1$ distance between two discrete probability measures $P$ and $Q$ from empirical data in a nonasymptotic and large alphabet setting. When $Q$ is known and one obtains $n$ samples from $P$, we show that for every $Q$, the minimax rate-optimal estimator with $n$ samples achieves performance comparable to that of the maximum likelihood estimator (MLE) with $n\ln n$ samples. When both $P$ and $Q$ are unknown, we construct minimax rate-optimal estimators whose worst case performance is essentially that of the known $Q$ case with $Q$ being uniform, implyi…
While efficient distribution learning is no doubt behind the groundbreaking success of diffusion modeling, its theoretical guarantees are quite limited. In this paper, we provide the first rigorous analysis on approximation and generalization abilities of diffusion modeling for well-known function spaces. The highlight of this paper is that when the true density function belongs to the Besov space and the empirical score matching loss is properly minimized, the generated data distribution achieves the nearly minimax optimal estimation rates in the total variation distance and in the Wasserstei…
We consider high-dimensional multivariate linear regression models, where the joint distribution of covariates and response variables is a multivariate normal distribution with a bandable covariance matrix. The main goal of this paper is to estimate the regression coefficient matrix, which is a function of the bandable covariance matrix. Although the tapering estimator of covariance has the minimax optimal convergence rate for the class of bandable covariances, we show that it has a sub-optimal convergence rate for the regression coefficient; that is, a minimax estimator for the class of banda…
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
We study the problem of estimating an unknown function from noisy data using shallow ReLU neural networks. The estimators we study minimize the sum of squared data-fitting errors plus a regularization term proportional to the squared Euclidean norm of the network weights. This minimization corresponds to the common approach of training a neural network with weight decay. We quantify the performance (mean-squared error) of these neural network estimators when the data-generating function belongs to the second-order Radon-domain bounded variation space. This space of functions was recently propo…
The asymptotic normality of the maximum likelihood estimator (MLE) under regularity conditions is a cornerstone of statistical theory. In this paper, we give explicit upper bounds on the distributional distance between the distribution of the MLE of a vector parameter, and the multivariate normal distribution. We work with possibly high-dimensional, independent but not necessarily identically distributed random vectors. In addition, we obtain explicit upper bounds even in cases where the MLE cannot be expressed analytically.
