Consider an i.i.d. sequence of random variables whose distribution f* lies in one of a nested family of models M_q, q>=1. The smallest index q* such that M_{q*} contains f* is called the model order. We establish strong consistency of the penalized likelihood order estimator in a general setting with penalties of order η(q) log log n, where η(q) is a dimensional quantity. Moreover, such penalties are shown to be minimal. In contrast to previous work, an a priori upper bound on the model order is not assumed. The results rely on a sharp characterization of the pathwise fluctuations of the generalized likelihood ratio statistic under entropy assumptions on the model classes. Our results are applied to the geometrically complex problem of location mixture order estimation, which is widely used but poorly understood.
The MDL two-part coding $ \textit{index of resolvability} $ provides a finite-sample upper bound on the statistical risk of penalized likelihood estimators over countable models. However, the bound does not apply to unpenalized maximum likelihood estimation or procedures with exceedingly small penalties. In this paper, we point out a more general inequality that holds for arbitrary penalties. In addition, this approach makes it possible to derive exact risk bounds of order $1/n$ for iid parametric models, which improves on the order $(\log n)/n$ resolvability bounds. We conclude by discussing …
This text is the rejoinder following the discussion of a survey paper about minimal penalties and the slope heuristics (Arlot, 2019. Minimal penalties and the slope heuristics: a survey. Journal de la SFDS). While commenting on the remarks made by the discussants, it provides two new results about the slope heuristics for model selection among a collection of projection estimators in least-squares fixed-design regression. First, we prove that the slope heuristics works even when all models are significantly biased. Second, when the noise is Gaussian with a general dependence structure, we comp…
The aim of this paper is to introduce an adaptive penalized estimator for identifying the true reduced parametric model under the sparsity assumption. In particular, we deal with the framework where the unpenalized estimator of the structural parameters needs simultaneously multiple rates of convergence (i.e. the so-called mixed-rates asymptotic behavior). We introduce a Bridge-type estimator by taking into account penalty functions involving $\ell^q$ norms $(0<q\leq 1)$. We prove that the proposed regularized estimator satisfies the oracle properties. Our approach is useful for the estimation…
Mean and variance estimation in high-dimensional heteroscedastic models with non-convex penalties
尽管异方差（非常数样本方差）在统计数据集中很普遍，但在高维统计文献中，它在很大程度上被忽略了。最近，研究表明，Lasso可以适应异方差误差，只需进行微小的算法修改（Belloni等人，2012; Gautier and Tsybakov，2013）.本文研究了高维稀疏参数下的线性均值模型和对数线性方差模型的异方差回归问题。在这项工作中，我们提出了估计方差后Lasso时尚，其次是加权最小二乘均值估计。这些步骤使用了Fan和Li（2001）中的非凸罚分，这使我们能够证明后Lasso方差和均值参数估计的预言性质。我们用实验来加强我们的理论发现。
Birg{é} and Massart proposed in 2001 the slope heuristics as a way to choose optimally from data an unknown multiplicative constant in front of a penalty. It is built upon the notion of minimal penalty, and it has been generalized since to some "minimal-penalty algorithms". This paper reviews the theoretical results obtained for such algorithms, with a self-contained proof in the simplest framework, precise proof ideas for further generalizations, and a few new results. Explicit connections are made with residual-variance estimators-with an original contribution on this topic, showing that for…
We propose a method for estimation in high-dimensional linear models with nominal categorical data. Our estimator, called SCOPE, fuses levels together by making their corresponding coefficients exactly equal. This is achieved using the minimax concave penalty on differences between the order statistics of the coefficients for a categorical variable, thereby clustering the coefficients. We provide an algorithm for exact and efficient computation of the global minimum of the resulting nonconvex objective in the case with a single variable with potentially many levels, and use this within a block…
We investigate whether in a distributed setting, adaptive estimation of a smooth function at the optimal rate is possible under minimal communication. It turns out that the answer depends on the risk considered and on the number of servers over which the procedure is distributed. We show that for the $L_\infty$-risk, adaptively obtaining optimal rates under minimal communication is not possible. For the $L_2$-risk, it is possible over a range of regularities that depends on the relation between the number of local servers and the total sample size.
Institute of Environmental and Agricultural Biology (X-BIO), University of Tyumen（秋明大学生态与农业生物学研究所（X-BIO））；University of Mississippi（密西西比大学）
We consider regression estimation with modified ReLU neural networks in which network weight matrices are first modified by a function $α$ before being multiplied by input vectors. We give an example of continuous, piecewise linear function $α$ for which the empirical risk minimizers over the classes of modified ReLU networks with $l_1$ and squared $l_2$ penalties attain, up to a logarithmic factor, the minimax rate of prediction of unknown $β$-smooth function.
We study the fundamental task of estimating the median of an underlying distribution from a finite number of samples, under pure differential privacy constraints. We focus on distributions satisfying the minimal assumption that they have a positive density at a small neighborhood around the median. In particular, the distribution is allowed to output unbounded values and is not required to have finite moments. We compute the exact, up-to-constant terms, statistical rate of estimation for the median by providing nearly-tight upper and lower bounds. Furthermore, we design a polynomial-time diffe…
