We study the problem of nonparametric estimation under $\bL_p$-loss, $p\in [1,\infty)$, in the framework of the convolution structure density model on $\bR^d$. This observation scheme is a generalization of two classical statistical models, namely density estimation under direct and indirect observations. In Part I the original pointwise selection rule from a family of "kernel-type" estimators is proposed. For the selected estimator, we prove an $\bL_p$-norm oracle inequality and several of its consequences. In Part II the problem of adaptive minimax estimation under $\bL_p$--loss over the scale of anisotropic Nikol'skii classes is addressed. We fully characterize the behavior of the minimax risk for different relationships between regularity parameters and norm indexes in the definitions of the functional class and of the risk. We prove that the selection rule proposed in Part I leads to the construction of an optimally or nearly optimally (up to logarithmic factor) adaptive estimator.
Many statistical estimation procedures lead to nonconvex optimization problems. Algorithms to solve these are often guaranteed to output a stationary point of the optimization problem. Oracle inequalities are an important theoretical instrument to asses the statistical performance of an estimator. Oracle results have focused on the theoretical properties of the uncomputable (global) minimum or maximum. In the present work a general framework used for convex optimization problems to derive oracle inequalities for stationary points is extended. A main new ingredient of these oracle inequalities …
This paper is a survey of recent results on the adaptive robust non parametric methods for the continuous time regression model with the semi - martingale noises with jumps. The noises are modeled by the Lévy processes, the Ornstein -- Uhlenbeck processes and semi-Markov processes. We represent the general model selection method and the sharp oracle inequalities methods which provide the robust efficient estimation in the adaptive setting. Moreover, we present the recent results on the improved model selection methods for the nonparametric estimation problems.
In this paper,we consider a high-dimensional statistical estimation problem in which the the number of parameters is comparable or larger than the sample size. We present a unified analysis of the performance guarantees of exponential weighted aggregation and penalized estimators with a general class of data losses and priors which encourage objects which conform to some notion of simplicity/complexity. More precisely, we show that these two estimators satisfy sharp oracle inequalities for prediction ensuring their good theoretical performances. We also highlight the differences between them. …
Inhomogeneous random graph models encompass many network models such as stochastic block models and latent position models. We consider the problem of statistical estimation of the matrix of connection probabilities based on the observations of the adjacency matrix of the network. Taking the stochastic block model as an approximation, we construct estimators of network connection probabilities -- the ordinary block constant least squares estimator, and its restricted version. We show that they satisfy oracle inequalities with respect to the block constant oracle. As a consequence, we derive op…
The Maximum Likelihood Estimator (MLE) serves an important role in statistics and machine learning. In this article, for i.i.d. variables, we obtain constant-specified and sharp concentration inequalities and oracle inequalities for the MLE only under exponential moment conditions. Furthermore, in a robust setting, the sub-Gaussian type oracle inequalities of the log-truncated maximum likelihood estimator are derived under the second-moment condition.
High-dimensional data have recently been analyzed because of data collection technology evolution. Although many methods have been developed to gain sparse recovery in the past two decades, most of these methods require selection of tuning parameters. As a consequence of this feature, results obtained with these methods heavily depend on the tuning. In this paper we study the theoretical properties of sign-constrained generalized linear models with convex loss function, which is one of the sparse regression methods without tuning parameters. Recent studies on this topic have shown that, in the…
We study the maximum likelihood estimator of density of $n$ independent observations, under the assumption that it is well approximated by a mixture with a large number of components. The main focus is on statistical properties with respect to the Kullback-Leibler loss. We establish risk bounds taking the form of sharp oracle inequalities both in deviation and in expectation. A simple consequence of these bounds is that the maximum likelihood estimator attains the optimal rate $((\log K)/n)^{1/2}$, up to a possible logarithmic correction, in the problem of convex aggregation when the number $K…
This paper presents an adaptive version of the Hill estimator based on Lespki's model selection method. This simple data-driven index selection method is shown to satisfy an oracle inequality and is checked to achieve the lower bound recently derived by Carpentier and Kim. In order to establish the oracle inequality, we derive non-asymptotic variance bounds and concentration inequalities for Hill estimators. These concentration inequalities are derived from Talagrand's concentration inequality for smooth functions of independent exponentially distributed random variables combined with three to…
This paper studies the non-asymptotic merits of the double $\ell_1$-regularized for heterogeneous overdispersed count data via negative binomial regressions. Under the restricted eigenvalue conditions, we prove the oracle inequalities for Lasso estimators of two partial regression coefficients for the first time, using concentration inequalities of empirical processes. Furthermore, derived from the oracle inequalities, the consistency and convergence rate for the estimators are the theoretical guarantees for further statistical inference. Finally, both simulations and a real data analysis demo…
In this paper for the first time the nonparametric autoregression estimation problem for the quadratic risks is considered. To this end we develop a new adaptive sequential model selection method based on the efficient sequential kernel estimators proposed by Arkoun and Pergamenshchikov (2016). Moreover, we develop a new analytical tool for general regression models to obtain the non asymptotic sharp or- acle inequalities for both usual quadratic and robust quadratic risks. Then, we show that the constructed sequential model selection proce- dure is optimal in the sense of oracle inequalities.
$\ell_1$-penalized quantile regression is widely used for analyzing high-dimensional data with heterogeneity. It is now recognized that the $\ell_1$-penalty introduces non-negligible estimation bias, while a proper use of concave regularization may lead to estimators with refined convergence rates and oracle properties as the signal strengthens. Although folded concave penalized $M$-estimation with strongly convex loss functions have been well studied, the extant literature on quantile regression is relatively silent. The main difficulty is that the quantile loss is piecewise linear: it is non…
We study various constraints and conditions on the true coefficient vector and on the design matrix to establish non-asymptotic oracle inequalities for the prediction error, estimation accuracy and variable selection for the Lasso estimator in high dimensional sparse regression models. We review results from the literature and we provide simpler and detailed derivation for several boundedness theorems. In addition, we complement the theory with illustrated examples.
We consider non parametric estimation problem for stochastic tomography regression model, i.e. we consider the estimation problem of function of multivariate variables (image) observed through its Radon transformation calculated with the random errors. For this problem we develop a new adaptive model selection method. By making use the Galtchouk and Pergamenshchikov approach we construct the model selection procedure for which we show a sharp non asymptotic oracle inequality for the both usual and robust quadratic risks, i.e. we show that the proposed procedure is optimal in the oracle inequal…
Sharp oracle inequalities for the prediction of a high-dimensional matrix
我们观察到$(X_i,Y_i)_{i=1}^n$，其中$Y_i$是实值输出，$X_i$是$m\times T$矩阵。我们观察到一个新的条目$X$，我们希望预测与之相关的输出$Y$。我们专注于高维设置，其中$m T \gg n$。这包括有噪声的矩阵完备问题，以及其他问题。我们考虑线性预测程序的基础上不同的惩罚，涉及几个规范的混合物：核规范，Frobenius规范和$\ell_1$规范。对于这些程序，我们证明了尖锐的甲骨文不等式，使用统计学习理论的观点。在我们的结果中，一个令人惊讶的事实是，收敛速度不直接依赖于$m$和$T$。该方法不需要通常考虑的未知矩阵的不相干条件和采样算子的等距约束条件。此外，我们的结果是第一个给这个问题的惩罚（如核范数惩罚）作为一个正则化算法的分析：我们的预言不等式证明，这些程序有一个预测精度接近确定性的预言之一，给定的正则化参数是精心选择的。
We give oracle inequalities on procedures which combines quantization and variable selection via a weighted Lasso $k$-means type algorithm. The results are derived for a general family of weights, which can be tuned to size the influence of the variables in different ways. Moreover, these theoretical guarantees are proved to adapt the corresponding sparsity of the optimal codebooks, if appropriate. Even if there is no sparsity assumption on the optimal codebooks, our procedure is proved to be close to a sparse approximation of the optimal codebooks, as has been done for the Generalized Linear …
We study the problem of estimating the one-point specification probabilities in non-necessary finite discrete random fields from partially observed independent samples. Our procedures are based on model selection by minimization of a penalized empirical criterion. The selected estimators satisfy sharp oracle inequalities in $L_2$-risk. We also obtain theoretical results on the slope heuristic for this problem, justifying the slope algorithm to calibrate the leading constant in the penalty. The practical performances of our methods are investigated in two simulation studies. We illustrate the u…
Through the direct study of the analysis estimator we derive oracle inequalities with fast and slow rates by adapting the arguments involving projections by Dalalyan, Hebiri and Lederer (2017). We then extend the theory to the square root analysis estimator. Finally, we focus on (square root) total variation regularized estimators on graphs and obtain constant-friendly rates, which, up to log-terms, match previous results obtained by entropy calculations. We also obtain an oracle inequality for the (square root) total variation regularized estimator over the cycle graph.
This note aims at presenting several new theoretical results for the compound Poisson point process, which follows the work of Zhang \emph{et al.} [Insurance~Math.~Econom.~59(2014), 325-336]. The first part provides a new characterization for a discrete compound Poisson point process (proposed by {Acz{é}l} [Acta~Math.~Hungar.~3(3)(1952), 219-224]), it extends the characterization of the Poisson point process given by Copeland and Regan [Ann.~Math.~(1936): 357-362]. Next, we derive some concentration inequalities for discrete compound Poisson point process (negative binomial random variable wit…
We study a set of regularization methods for high-dimensional linear regression models. These penalized estimators have the square root of the residual sum of squared errors as loss function, and any weakly decomposable norm as penalty function. This fit measure is chosen because of its property that the estimator does not depend on the unknown standard deviation of the noise. On the other hand, a generalized weakly decomposable norm penalty is very useful in being able to deal with different underlying sparsity structures. We can choose a different sparsity inducing norm depending on how we w…
The performance of Least Squares (LS) estimators is studied in isotonic, unimodal and convex regression. Our results have the form of sharp oracle inequalities that account for the model misspecification error. In isotonic and unimodal regression, the LS estimator achieves the nonparametric rate $n^{-2/3}$ as well as a parametric rate of order $k/n$ up to logarithmic factors, where $k$ is the number of constant pieces of the true parameter. In univariate convex regression, the LS estimator satisfies an adaptive risk bound of order $q/n$ up to logarithmic factors, where $q$ is the number of aff…
We obtain estimation error rates and sharp oracle inequalities for regularization procedures of the form \begin{equation*} \hat f \in argmin_{f\in F}\left(\frac{1}{N}\sum_{i=1}^N\ell(f(X_i), Y_i)+λ\|f\|\right) \end{equation*} when $\|\cdot\|$ is any norm, $F$ is a convex class of functions and $\ell$ is a Lipschitz loss function satisfying a Bernstein condition over $F$. We explore both the bounded and subgaussian stochastic frameworks for the distribution of the $f(X_i)$'s, with no assumption on the distribution of the $Y_i$'s. The general results rely on two main objects: a complexity functi…
The abundance of high-dimensional data in the modern sciences has generated tremendous interest in penalized estimators such as the lasso, scaled lasso, square-root lasso, elastic net, and many others. In this paper, we establish a general oracle inequality for prediction in high-dimensional linear regression with such methods. Since the proof relies only on convexity and continuity arguments, the result holds irrespective of the design matrix and applies to a wide range of penalized estimators. Overall, the bound demonstrates that generic estimators can provide consistent prediction with any …
