We generalize to tree graphs obtained by connecting path graphs an oracle result obtained for the Fused Lasso over the path graph. Moreover we show that it is possible to substitute in the oracle inequality the minimum of the distances between jumps by their harmonic mean. In doing so we prove a lower bound on the compatibility constant for the total variation penalty. Our analysis leverages insights obtained for the path graph with one branch to understand the case of more general tree graphs.
As a side result, we get insights into the irrepresentable condition for such tree graphs.
The Convex Gaussian Min-Max Theorem (CGMT) has emerged as a prominent theoretical tool for analyzing the precise stochastic behavior of various statistical estimators in the so-called high dimensional proportional regime, where the sample size and the signal dimension are of the same order. However, a well recognized limitation of the existing CGMT machinery rests in its stringent requirement on the exact Gaussianity of the design matrix, therefore rendering the obtained precise high dimensional asymptotics largely a specific Gaussian theory in various important statistical models. This paper …
A compositional tree refers to a tree structure on a set of random variables where each random variable is a node and composition occurs at each non-leaf node of the tree. As a generalization of compositional data, compositional trees handle more complex relationships among random variables and appear in many disciplines, such as brain imaging, genomics and finance. We consider the problem of sparse regression on data that are associated with a compositional tree and propose a transformation-free tree-based regularized regression method for component selection. The regularization penalty is de…
In this paper, we consider matrix completion with absolute deviation loss and obtain an estimator of the median matrix. Despite several appealing properties of median, the non-smooth absolute deviation loss leads to computational challenge for large-scale data sets which are increasingly common among matrix completion problems. A simple solution to large-scale problems is parallel computing. However, embarrassingly parallel fashion often leads to inefficient estimators. Based on the idea of pseudo data, we propose a novel refinement step, which turns such inefficient estimators into a rate (ne…
We present a general framework for studying regularized estimators; such estimators are pervasive in estimation problems wherein "plug-in" type estimators are either ill-defined or ill-behaved. Within this framework, we derive, under primitive conditions, consistency and a generalization of the asymptotic linearity property. We also provide data-driven methods for choosing tuning parameters that, under some conditions, achieve the aforementioned properties. We illustrate the scope of our approach by presenting a wide range of applications.
The asymptotic optimality (a.o.) of various hyper-parameter estimators with different optimality criteria has been studied in the literature for regularized least squares regression problems. The estimators include e.g., the maximum (marginal) likelihood method, $C_p$ statistics, and generalized cross validation method, and the optimality criteria are based on e.g., the inefficiency, the expectation inefficiency and the risk. In this paper, we consider the regularized least squares regression problems with fixed number of regression parameters, choose the optimality criterion based on the risk…
We study Empirical Risk Minimizers (ERM) and Regularized Empirical Risk Minimizers (RERM) for regression problems with convex and $L$-Lipschitz loss functions. We consider a setting where $|\cO|$ malicious outliers contaminate the labels. In that case, under a local Bernstein condition, we show that the $L_2$-error rate is bounded by $ r_N + AL |\cO|/N$, where $N$ is the total number of observations, $r_N$ is the $L_2$-error rate in the non-contaminated setting and $A$ is a parameter coming from the local Bernstein condition. When $r_N$ is minimax-rate-optimal in a non-contaminated setting, th…
In this paper, we give a tutorial on asymptotic properties of the Least Square (LS) and Regularized Least Squares (RLS) estimators for the finite impulse response model with filtered white noise inputs. We provide three perspectives: the almost sure convergence, the convergence in distribution and the boundedness in probability. On one hand, these properties deepen our understanding of the LS and RLS estimators. On the other hand, we can use them as tools to investigate asymptotic properties of other estimators, such as various hyper-parameter estimators.
The aim of this paper is to introduce an adaptive penalized estimator for identifying the true reduced parametric model under the sparsity assumption. In particular, we deal with the framework where the unpenalized estimator of the structural parameters needs simultaneously multiple rates of convergence (i.e. the so-called mixed-rates asymptotic behavior). We introduce a Bridge-type estimator by taking into account penalty functions involving $\ell^q$ norms $(0<q\leq 1)$. We prove that the proposed regularized estimator satisfies the oracle properties. Our approach is useful for the estimation…
The Highest Dimensional Stochastic Blockmodel with a Regularized Estimator
在随机网络的高维随机块模型中，簇（或块）的数目K随着节点的数目N而增长。两个先前的研究已经检查了谱聚类和最大似然估计器在高维模型下的统计估计性能;这些结果都不允许K增长快于N^{1/2}。我们研究了一个模型，其中忽略对数项，K可以与N成比例增长。由于簇的数量必须小于节点的数量，没有合理的模型允许K增长得更快;因此，我们的渐近结果是“最高”维。为了将渐近设置推向这个极端，我们做出了额外的假设，这些假设是由体质人类学的经验观察（Dunbar，1992）和对大规模经验网络的深入研究（Leskovec et al 2008）所激发的。此外，我们开发了一个正则化的最大似然估计，利用这些见解，我们证明，在某些条件下，正则化估计错误聚类的节点的比例收敛到零。这是第一篇论文明确介绍和证明统计正则化的优势，在参数形式的网络分析。
From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators
Department of Economics, University of Pennsylvania（宾夕法尼亚大学经济学系）；Department of Economics, University of Oxford（牛津大学经济学系）；Department of Electrical Engineering and Computer Science, MIT（麻省理工学院电气工程与计算机科学系）
Robust learning and complexity dependent bounds for regularized problems
我们研究正则化经验风险最小化（RERM）和最小最大均值（MEN）估计，其中正则化函数$ϕ(\cdot)$是一个甚至凸函数。我们得到的$L_2$ -估计误差和超额风险的界限，依赖于$ϕ(f^*)$，其中$f^*$是一个类$F$的风险最小。估计是基于损失函数，这是Lipschitz和凸。RERM的结果来自弱假设的输出和类$\{ (f-f^*)(X), f \in F \}$的亚高斯假设。在一个封闭的设置中，离群值可能会破坏数据集，并且类$\{ (f-f^*)(X), f \in F \}$只应该满足弱矩假设，放松RERM所需的亚高斯和i.i.d假设，对于极小极大极小估计量也显示了类似的结果。本文在弱局部Bernstein假设的基础上，分析了具有Lipschitz和凸损失函数的RERM和minmax估计量。我们得到了两个“Meta定理”，我们使用它来研究弹性网络正则化的线性估计。我们还研究了支持向量机（SVM），其中没有亚高斯假设是必需的，当目标$Y$可以重尾，改善现有的文献。
We establish adaptive results for trend filtering: least squares estimation with a penalty on the total variation of $(k-1)^{\rm th}$ order differences. Our approach is based on combining a general oracle inequality for the $\ell_1$-penalized least squares estimator with "interpolating vectors" to upper-bound the "effective sparsity". This allows one to show that the $\ell_1$-penalty on the $k^{\text{th}}$ order differences leads to an estimator that can adapt to the number of jumps in the $(k-1)^{\text{th}}$ order differences of the underlying signal or an approximation thereof. We show the r…
A unifying approach for doubly-robust $\ell_1$ regularized estimation of causal contrasts
We consider inference about a scalar parameter under a non-parametric model based on a one-step estimator computed as a plug in estimator plus the empirical mean of an estimator of the parameter's influence function . We focus on a class of parameters that have influence function which depends on two infinite dimensional nuisance functions and such that the bias of the one-step estimator of the parameter of interest is the expectation of the product of the estimation errors of the two nuisance functions. Our class includes many important treatment effect contrasts of interest in causal inferen…
Neural networks have become standard tools in the analysis of data, but they lack comprehensive mathematical theories. For example, there are very few statistical guarantees for learning neural networks from data, especially for classes of estimators that are used in practice or at least similar to such. In this paper, we develop a general statistical guarantee for estimators that consist of a least-squares term and a regularizer. We then exemplify this guarantee with $\ell_1$-regularization, showing that the corresponding prediction error increases at most sub-linearly in the number of layers…
New upper bounds are developed for the $L_2$ distance between $ξ/\text{Var}[ξ]^{1/2}$ and linear and quadratic functions of $z\sim N(0,I_n)$ for random variables of the form $ξ=bz^\top f(z) - \text{div} f(z)$. The linear approximation yields a central limit theorem when the squared norm of $f(z)$ dominates the squared Frobenius norm of $\nabla f(z)$ in expectation. Applications of this normal approximation are given for the asymptotic normality of de-biased estimators in linear regression with correlated design and convex penalty in the regime $p/n \le γ$ for constant $γ\in(0,{\infty})$. For t…
