We obtain concentration and large deviation for the sums of independent and identically distributed random variables with heavy-tailed distributions. Our concentration results are concerned with random variables whose distributions satisfy $\mathbb{P}(X>t) \leq {\rm e}^{- I(t)}$, where $I: \mathbb{R} \rightarrow \mathbb{R}$ is an increasing function and $I(t)/t \rightarrow α\in [0, \infty)$ as $t \rightarrow \infty$. Our main theorem can not only recover some of the existing results, such as the concentration of the sum of subWeibull random variables, but it can also produce new results for the sum of random variables with heavier tails. We show that the concentration inequalities we obtain are sharp enough to offer large deviation results for the sums of independent random variables as well. Our analyses which are based on standard truncation arguments simplify, unify and generalize the existing results on the concentration and large deviation of heavy-tailed random variables.
Northwestern University（西北大学）；University of Washington（华盛顿大学）
The empirical success of deep learning is often attributed to SGD's mysterious ability to avoid sharp local minima in the loss landscape, as sharp minima are known to lead to poor generalization. Recently, empirical evidence of heavy-tailed gradient noise was reported in many deep learning tasks, and it was shown in Şimşekli (2019a,b) that SGD can escape sharp local minima under the presence of such heavy-tailed gradient noise, providing a partial solution to the mystery. In this work, we analyze a popular variant of SGD where gradients are truncated above a fixed threshold. We show that it ac…
We study deviation of U-statistics when samples have heavy-tailed distribution so the kernel of the U-statistic does not have bounded exponential moments at any positive point. We obtain an exponential upper bound for the tail of the U-statistics which clearly denotes two regions of tail decay, the first is a Gaussian decay and the second behaves like the tail of the kernel. For several common U-statistics, we also show the upper bound has the right rate of decay as well as sharp constants by obtaining rough logarithmic limits which in turn can be used to develop LDP for U-statistics. In spite…
University of California, San Diego（加利福尼亚大学圣地亚哥分校）
We give concentration bounds for martingales that are uniform over finite times and extend classical Hoeffding and Bernstein inequalities. We also demonstrate our concentration bounds to be optimal with a matching anti-concentration inequality, proved using the same method. Together these constitute a finite-time version of the law of the iterated logarithm, and shed light on the relationship between it and the central limit theorem.
We show an extension of Sanov's theorem on large deviations, controlling the tail probabilities of i.i.d. random variables with matching concentration and anti-concentration bounds. This result has a general scope, applies to samples of any size, and has a short information-theoretic proof using elementary techniques.
Recent development in high-dimensional statistical inference has necessitated concentration inequalities for a broader range of random variables. We focus on sub-Weibull random variables, which extend sub-Gaussian or sub-exponential random variables to allow heavy-tailed distributions. This paper presents concentration inequalities for independent sub-Weibull random variables with finite Generalized Bernstein-Orlicz norms, providing generalized Bernstein's inequalities and Rosenthal-type moment bounds. The tightness of the proposed bounds is shown through lower bounds of the concentration ineq…
Constant-specified and exponential concentration inequalities play an essential role in the finite-sample theory of machine learning and high-dimensional statistics area. We obtain sharper and constants-specified concentration inequalities for the sum of independent sub-Weibull random variables, which leads to a mixture of two tails: sub-Gaussian for small deviations and sub-Weibull for large deviations from the mean. These bounds are new and improve existing bounds with sharper constants. In addition, a new sub-Weibull parameter if the italic should be retained. Please check the whole text. i…
This paper investigates the decay rate of the probability that the row sum of a triangular array of truncated heavy tailed random variables is larger than an integer (k) times the truncating threshold, as both - the number of summands and the threshold go to infinity. The method of attack for this problem is significantly different from the one where k is not an integer, and requires much sharper estimates.
We prove concentration inequalities for $f\left( X\right) $ about its median, where $X$ is a random vector in $\mathbb{R}^n$ with independent heavy tailed coordinates of Weibull or power type, and $f:\mathbb{R}^n\rightarrow\mathbb{R}$ is a locally Lipschitz function. This paper is part of a series of four papers, Part I, Part II and two supporting papers. It can be read independently of Part I.
In this paper, we explore some links between transforms derived by Stein's method and concentration inequalities. In particular, we show that the stochastic domination of the zero bias transform of a random variable is equivalent to sub-Gaussian concentration. For this purpose a new stochastic order is considered. In a second time, we study the case of functions of slightly dependent light-tailed random variables. We are able to recover the famous McDiarmid type of concentration inequality for functions with the bounded difference property. Additionally, we obtain new concentration bounds when…
Sub-Gaussian estimates for the natural random walk is typical of many regular fractal graphs. Subordination shows that there exist heavy tailed jump processes whose jump indices are greater than or equal to two. However, the existing machinery used to prove heat kernel bounds for such heavy tailed random walks fail in this case. In this work we extend Davies' perturbation method to obtain transition probability bounds for these anomalous heavy tailed random walks. We prove global upper and lower bounds on the transition probability density that are sharp up to constants. An important feature o…
Considering random matrix $X \in \mathcal M_{p,n}$ with independent columns satisfying the convex concentration properties issued from a famous theorem of Talagrand, we express the linear concentration of the resolvent $Q = (I_p - \frac{1}{n}XX^T) ^{-1}$ around a classical deterministic equivalent with a good observable diameter for the nuclear norm. The general proof relies on a decomposition of the resolvent as a series of powers of $X$.
Consider the random set system of {1,2,...,n}, where each subset in the power set is chosen independently with probability p. A set H is said to be a hitting set if it intersects each chosen set. The second moment method is used to exhibit the sharp concentration of the minimal size of H for a variety of values of p.
We study one-dimensional functional inequalities of the type of Poincaré, logarithmic Sobolev and Wirtinger, with weight, for probability densities with polynomial tails. As main examples, we obtain sharp inequalities satisfied by inverse Gamma densities, taking values on $R_+$, and Cauchy-type densities, taking values on $R$. In this last case, we improve the result obtained by Bobkov and Ledoux in 2009 by introducing a better weight function in the logarithmic Sobolev inequality. The results are obtained by resorting to Fokker-Planck type equations which possess these densities as steady sta…
Motivated by the study of the time evolution of random dynamical systems arising in a vast variety of domains --- ranging from physics to ecology ---, we establish conditions for the occurrence of a non-trivial asymptotic behaviour for these systems in the absence of an ellipticity condition. More precisely, we classify these systems according to their type and --- in the recurrent case --- provide with sharp conditions quantifying the nature of recurrence by establishing which moments of passage times exist and which do not exist. The problem is tackled by mapping the random dynamical systems…
We prove that the largest and smallest root in modulus of random Kac polynomials have a non-universal behavior. They do not converge towards the edge of the support of the limiting distribution of the zeros. This non-universality is surprising as the large deviation principle for the empirical measure is universal. This is in sharp contrast with random matrix theory where the large deviation principle is non-universal but the fluctuations of the largest eigenvalue are universal. We show that the modulus of the largest zero is heavy tailed, with a number of finite moments bounded from above by …
Let $W_n= \frac{1}{\sqrt n} M_n$ be a Wigner matrix whose entries have vanishing third moment, normalized so that the spectrum is concentrated in the interval $[-2,2]$. We prove a concentration bound for $N_I = N_I(W_n)$, the number of eigenvalues of $W_n$ in an interval $I$. Our result shows that $N_I$ decays exponentially with standard deviation at most $O(\log^{O(1)} n)$. This is best possible up to the constant exponent in the logarithmic term. As a corollary, the bulk eigenvalues are localized to an interval of width $O(\log^{O(1)} n/n)$; again, this is optimal up to the exponent. These r…
For an ergodic Brownian diffusion with invariant measure $ν$, we consider a sequence of empirical distributions ($ν$n) n$\ge$1 associated with an approximation scheme with decreasing time step ($γ$n) n$\ge$1 along an adapted regular enough class of test functions f such that f --$ν$(f) is a coboundary of the infinitesimal generator A. Denote by $σ$ the diffusion coefficient and $Φ$ the solution of the Poisson equation A$Φ$ = f -- $ν$(f). When the square norm of |$σ$ * $Φ$| 2 lies in the same coboundary class as f , we establish sharp non-asymptotic concentration bounds for suitable normalizati…
For a large class of random matrices $A$ with i.i.d. entries we show that the $\ell_1$-quotient property holds with probability exponentially close to 1. In contrast to previous results, our analysis does not require concentration of the entrywise distributions. We provide a unified proof that recovers corresponding previous results for (sub-)Gaussian and Weibull distributions. Our findings generalize known results on the geometry of random polytopes, providing lower bounds on the size of the largest Euclidean ball contained in the centrally symmetric polytope spanned by the columns of $A$. At…
We give a shorter proof of Kanter's (1976) sharp Bessel function bound for concentrations of sums of independent symmetric random vectors. We provide sharp upper bounds for the sum of modified Bessel functions $I_0(x)+I_1(x)$, which might be of independent interest. Corollaries improve concentration or smoothness bounds for sums of independent random variables due to Cekanavicius & Roos (2006), Roos (2005), Barbour & Xia 1999), and Le Cam (1986).
We study the recurrence/transience phase transition for Markov chains on $\mathbb{R}_+$, $\mathbb{R}$, and $\mathbb{R}^2$ whose increments have heavy tails with exponent in $(1,2)$ and asymptotically zero mean. This is the infinite-variance analogue of the classical Lamperti problem. On $\mathbb{R}_+$, for example, we show that if the tail of the positive increments is about $c y^{-α}$ for an exponent $α\in (1,2)$ and if the drift at $x$ is about $b x^{-γ}$, then the critical regime has $γ= α-1$ and recurrence/transience is determined by the sign of $b + cπ\textrm{cosec} (πα)$. On $\mathbb{R}$…
This work is denoted to studying the tail behavior of Cox-Ingersoll-Ross (CIR) processes with regime-switching. One essential difference shown in this work between CIR process with regime-switching and without regime-switching is that the stationary distribution for CIR process with regime-switching could be heavy-tailed. Our results provide a theoretical evidence of the existence of regime-switching for interest rates model based on its heavy-tailed empirical evidence. In this work, we first provide sharp criteria to justify the existence of stationary distribution for the CIR process with re…
We study the asymptotic behavior of the principal eigenvector and eigenvalue of the random conductance Laplacian in a large domain of $\mathbb{Z}^d$ ($d\geq 2$) with zero Dirichlet condition. We assume that the conductances $w$ are positive i.i.d. random variables, which fulfill certain regularity assumptions near zero. If $γ=\sup \{ q\geq 0\colon \mathbb{E} [w^{-q}]<\infty \}<1/4$, then we show that for almost every environment the principal Dirichlet eigenvector asymptotically concentrates in a single site and the corresponding eigenvalue scales subdiffusively. The threshold $γ_{\rm c} = 1/4…
This paper focuses on the non-asymptotic concentration of the heteroskedastic Wishart-type matrices. Suppose $Z$ is a $p_1$-by-$p_2$ random matrix and $Z_{ij} \sim N(0,σ_{ij}^2)$ independently, we prove the expected spectral norm of Wishart matrix deviations (i.e., $\mathbb{E} \left\|ZZ^\top - \mathbb{E} ZZ^\top\right\|$) is upper bounded by \begin{equation*} \begin{split} (1+ε)\left\{2σ_Cσ_R + σ_C^2 + Cσ_Rσ_*\sqrt{\log(p_1 \wedge p_2)} + Cσ_*^2\log(p_1 \wedge p_2)\right\}, \end{split} \end{equation*} where $σ_C^2 := \max_j \sum_{i=1}^{p_1}σ_{ij}^2$, $σ_R^2 := \max_i \sum_{j=1}^{p_2}σ_{ij}^2$ …
We establish sharp exponential deviation estimates of the information content as well as a sharp bound on the varentropy for the class of convex measures on Euclidean spaces. This generalizes a similar development for log-concave measures in the recent work of Fradelizi, Madiman and Wang (2016). In particular, our results imply that convex measures in high dimensions are concentrated in an annulus between two convex sets (as in the log-concave case) despite their possibly having much heavier tails. Various tools and consequences are developed, including a sharp comparison result for Rényi entr…
We investigate concentration properties of spectral measures of Hermitian random matrices with partially dependent entries. More precisely, let $X_n$ be a Hermitian random matrix of size $n\times n$ that can be split into independent blocks of the size at most $d_n=o(n^2)$. We prove that under some mild conditions on the distribution of the entries of $X_n$, the empirical spectral measure of $X_n$ concentrates around its mean. The main theorem is a strengthening of a recent result by Kemp and Zimmerman, where the size of blocks grows as $o(\log n)$. As an application, we are able to upgrade th…
