While effective concentration inequalities for suprema of empirical processes exist under boundedness or strict tail assumptions, no comparable results have been available under considerably weaker assumptions. In this paper, we derive concentration inequalities assuming only low moments for an envelope of the empirical process. These concentration inequalities are beneficial even when the envelope is much larger than the single functions under consideration.
Let $\mathcal{F}$ be a class of measurable functions on a measurable space $(S,\mathcal{S})$ with values in $[0,1]$ and let \[P_n=n^{-1}\sum_{i=1}^nδ_{X_i}\] be the empirical measure based on an i.i.d. sample $(X_1,...,X_n)$ from a probability distribution $P$ on $(S,\mathcal{S})$. We study the behavior of suprema of the following type: \[\sup_{r_n<σ_Pf\leq δ_n}\frac{|P_nf-Pf|}{ϕ(σ_Pf)},\] where $σ_Pf\ge\operatorname {Var}^{1/2}_Pf$ and $ϕ$ is a continuous, strictly increasing function with $ϕ(0)=0$. Using Talagrand's concentration inequality for empirical processes, we establish concentration…
We explore the applications of our previously established likelihood-ratio method for deriving concentration inequalities for a wide variety of univariate and multivariate distributions. New concentration inequalities for various distributions are developed without the idea of minimizing moment generating functions.
We derive concentration inequalities for maxima of empirical processes associated with Poisson point processes. The proofs are based on a careful application of Ledoux's entropy method. We demonstrate the utility of the obtained concentration inequalities by application to adaptive intensity estimation.
We prove a Bennett-type concentration bound for suprema of empirical processes based on sampling without replacement and a corresponding bound in the case of an arbitrary Hoeffding statistics. We improve on the previous results of such type, providing a sharper concentration profile.
Matrix concentration inequalities give bounds for the spectral-norm deviation of a random matrix from its expected value. These results have a weak dimensional dependence that is sometimes, but not always, necessary. This paper identifies one of the sources of the dimensional term and exploits this insight to develop sharper matrix concentration inequalities. In particular, this analysis delivers two refinements of the matrix Khintchine inequality that use information beyond the matrix variance to reduce or eliminate the dimensional dependence.
We investigate links between the so-called Stein's density approach in dimension one and some functional and concentration inequalities. We show that measures having a finite first moment and a density with connected support satisfy a weighted Poincaré inequality with the weight being the Stein kernel, that indeed exists and is unique in this case. Furthermore, we prove weighted log-Sobolev and asymmetric Brascamp-Lieb type inequalities related to Stein kernels. We also show that existence of a uniformly bounded Stein kernel is sufficient to ensure a positive Cheeger isoperimetric constant. Th…
This paper presents concentration inequalities and laws of large numbers under weak assumptions of irrelevance, expressed through lower and upper expectations. The results are variants and extensions of De Cooman and Miranda's recent inequalities and laws of large numbers. The proofs indicate connections between concepts of irrelevance for lower/upper expectations and the standard theory of martingales.
Our goal is to establish large deviations and concentration inequalities for the maximum likelihood estimator of the drift parameter of the Ornstein-Uhlenbeck process without tears. We propose a new strategy to establish large deviation results which allows us, via a suitable transformation, to circumvent the classical difficulty of non-steepness. Our approach holds in the stable case where the process is positive recurrent as well as in the unstable and explosive cases where the process is respectively null recurrent and transient. Notwithstanding of this trichotomy, we also provide new conce…
Let $\pa{X_{t}}_{t\in T}$ be a family of real-valued centered random variables indexed by a countable set $T$. In the first part of this paper, we establish exponential bounds for the deviation probabilities of the supremum $Z=\sup_{t\in T}X_{t}$ by using the generic chaining device introduced in Talagrand (2005). Compared to concentration-type inequalities, these bounds offer the advantage to hold under weaker conditions on the family $\pa{X_{t}}_{t\in T}$. The second part of the paper is oriented towards statistics. We consider the regression setting $Y=f+\eps$ where $f$ is an unknown vector…
Louisiana State University（路易斯安那州立大学）；Southern University and A&M College（南方大学及农工学院）
We derive simple concentration inequalities for bounded random vectors, which generalize Hoeffding's inequalities for bounded scalar random variables. As applications, we apply the general results to multinomial and Dirichlet distributions to obtain multivariate concentration inequalities.
Initially motivated by the study of the non-asymptotic properties of non-parametric tests based on permutation methods, concentration inequalities for uniformly permuted sums have been largely studied in the literature. Recently, Delyon et al. proved a new Bernstein-type concentration inequality based on martingale theory. This work presents a new proof of this inequality based on the fundamental inequalities for random permutations of Talagrand. The idea is to first obtain a rough inequality for the square root of the permuted sum, and then, iterate the previous analysis and plug this first i…
In this report, we aim to exemplify concentration inequalities and provide easy to understand proofs for it. Our focus is on the inequalities which are helpful in the design and analysis of machine learning algorithms.
In this paper, we explore some links between transforms derived by Stein's method and concentration inequalities. In particular, we show that the stochastic domination of the zero bias transform of a random variable is equivalent to sub-Gaussian concentration. For this purpose a new stochastic order is considered. In a second time, we study the case of functions of slightly dependent light-tailed random variables. We are able to recover the famous McDiarmid type of concentration inequality for functions with the bounded difference property. Additionally, we obtain new concentration bounds when…
This note aims at presenting several new theoretical results for the compound Poisson point process, which follows the work of Zhang \emph{et al.} [Insurance~Math.~Econom.~59(2014), 325-336]. The first part provides a new characterization for a discrete compound Poisson point process (proposed by {Acz{é}l} [Acta~Math.~Hungar.~3(3)(1952), 219-224]), it extends the characterization of the Poisson point process given by Copeland and Regan [Ann.~Math.~(1936): 357-362]. Next, we derive some concentration inequalities for discrete compound Poisson point process (negative binomial random variable wit…
Recent development in high-dimensional statistical inference has necessitated concentration inequalities for a broader range of random variables. We focus on sub-Weibull random variables, which extend sub-Gaussian or sub-exponential random variables to allow heavy-tailed distributions. This paper presents concentration inequalities for independent sub-Weibull random variables with finite Generalized Bernstein-Orlicz norms, providing generalized Bernstein's inequalities and Rosenthal-type moment bounds. The tightness of the proposed bounds is shown through lower bounds of the concentration ineq…
This note describes non-asymptotic variance and tail bounds for order statistics of samples of independent identically distributed random variables. Those bounds are checked to be asymptotically tight when the sampling distribution belongs to a maximum domain of attraction. If the sampling distribution has non-decreasing hazard rate (this includes the Gaussian distribution), we derive an exponential Efron-Stein inequality for order statistics: an inequality connecting the logarithmic moment generating function of centered order statistics with exponential moments of Efron-Stein (jackknife) est…
Concentration inequalities for strong laws and laws of the iterated logarithm
Department of Mathematics, London School of Economics and Political Science（伦敦政治经济学院数学系）；Department of Statistics and The Miller Institute, University of California, Berkeley（加州大学伯克利分校统计系及米勒研究所）
For any nonnegative Borel-measurable function f such that f(x)=0 if and only if x=0, the best constant c_f in the inequality E f(X-E X) \leq c_f E f(X) for all random variables X with a finite mean is obtained. Properties of the constant c_f in the case when f=|.|^p are studied. Applications to concentration of measure in the form of Rosenthal-type bounds on the moments of separately Lipschitz functions on product spaces are given.
We derive concentration inequalities for empirical means $\frac{1}{t} \int_0^t f(X_s) ds$ where $X_s$ is an irreducible Markov jump process on a finite state space and $f$ some observable. Using a Feynman-Kac semigroup we first derive a general concentration inequality. Then, based on this inequality we derive further concentration inequalities. Hereby we use three different methods; perturbation theory, Poincaré inequalities and information inequalities. We also obtain a Bernstein type concentration inequality.
We establish concentration inequalities in the class of ultra log-concave distributions. In particular, we show that ultra log-concave distributions satisfy Poisson concentration bounds. As an application, we derive concentration bounds for the intrinsic volumes of a convex body, which generalizes and improves a result of Lotz, McCoy, Nourdin, Peccati, and Tropp (2019).
This paper presents compact notations for concentration inequalities and convenient results to streamline probabilistic analysis. The new expressions describe the typical sizes and tails of random variables, allowing for simple operations without heavy use of inessential constants. They bridge classical asymptotic notations and modern non-asymptotic tail bounds together. Examples of different kinds demonstrate their efficacy.
We consider a random variable $X$ that takes values in a (possibly infinite-dimensional) topological vector space $\mathcal{X}$. We show that, with respect to an appropriate "normal distance" on $\mathcal{X}$, concentration inequalities for linear and non-linear functions of $X$ are equivalent. This normal distance corresponds naturally to the concentration rate in classical concentration results such as Gaussian concentration and concentration on the Euclidean and Hamming cubes. Under suitable assumptions on the roundness of the sets of interest, the concentration inequalities so obtained are…
Concentration Inequalities for Statistical Inference
Center for Statistical Sciences, Peking University（北京大学统计科学中心）；Department of Mathematics, Faculty of Science and Technology, University of Macau（澳门大学科技学院数学系）
This paper explores the connection between a generalized Riesz electric energy and norms on the set of probability measures defined in terms of duality. We derive functional inequalities linking these two notions, recovering and generalizing existing Coulomb transport inequalities. We then use them to prove concentration of measure around the equilibrium and thermal equilibrium measures. Finally, we leverage these concentration inequalities to obtain Moser-Trudinger-type inequalities, which may also be interpreted as bounds on the Laplace transform of fluctuations.
We introduce two new concepts designed for the study of empirical processes. First, we introduce a new Orlicz norm which we call the Bernstein-Orlicz norm. This new norm interpolates sub-Gaussian and sub-exponential tail behavior. In particular, we show how this norm can be used to simplify the derivation of deviation inequalities for suprema of collections of random variables. Secondly, we introduce chaining and generic chaining along a tree. These simplify the well-known concepts of chaining and generic chaining. The supremum of the empirical process is then studied as a special case. We sho…
This paper studies concentration inequalities for functions of locally dependent random variables. We show that the usual definition of local dependence does not imply concentration for general Hamming Lipschitz functions. We define hypergraph dependence, which is a special case of local dependence, and show that it implies concentration if the maximal neighborhood size is small. We prove concentration in Hamming distance, Talagrand distance, and for self-bounding functions of a particular type under this dependence structure.
In this work, we will generalize the moment generating function to Riesz spaces. We will derive some of its properties and use it to prove concentration inequalities on Riesz spaces.
Let $\mathbf{W}=(W_1,W_2,...,W_k)$ be a random vector with nonnegative coordinates having nonzero and finite variances. We prove concentration inequalities for $\mathbf{W}$ using size biased couplings that generalize the previous univariate results. Two applications on local dependence and counting patterns are provided.
We introduce a version of Stein's method for proving concentration and moment inequalities in problems with dependence. Simple illustrative examples from combinatorics, physics, and mathematical statistics are provided.
We propose new concentration inequalities for self-normalized martingales. The main idea is to introduce a suitable weighted sum of the predictable quadratic variation and the total quadratic variation of the martingale. It offers much more flexibility and allows us to improve previous concentration inequalities. Statistical applications on autoregressive process, internal diffusion-limited aggregation process, and online statistical learning are also provided.
In this work we derive multi-level concentration inequalities for polynomial functions in independent random variables with a $α$-sub-exponential tail decay. A particularly interesting case is given by quadratic forms $f(X_1, \ldots, X_n) = \langle X,A X \rangle$, for which we prove Hanson-Wright-type inequalities with explicit dependence on various norms of the matrix $A$. A consequence of these inequalities is a two-level concentration inequality for quadratic forms in $α$-sub-exponential random variables, such as quadratic Poisson chaos. We provide various applications of these inequalities…
We consider a Boolean model $Z$ driven by a Poisson particle process $η$ on a metric space $\mathbb{Y}$. We study the random variable $ρ(Z)$, where $ρ$ is a (deterministic) measure on $\mathbb{Y}$. Due to the interaction of overlapping particles, the distribution of $ρ(Z)$ cannot be described explicitly. In this note we derive concentration inequalities for $ρ(Z)$. To this end we first prove two concentration inequalities for functions of a Poisson process on a general phase space.
Non-Gaussian concentration estimates are obtained for invariant probability measures of reversible Markov processes. We show that the functional inequalities approach combined with a suitable Lyapunov condition allows us to circumvent the classical Lipschitz assumption of the observables. Our method is general and covers diffusions as well as pure-jump Markov processes on unbounded spaces.
We show that any probability measure satisfying a Matrix Poincaré inequality with respect to some reversible Markov generator satisfies an exponential matrix concentration inequality depending on the associated matrix carré du champ operator. This extends to the matrix setting a classical phenomenon in the scalar case. Moreover, the proof gives rise to new matrix trace inequalities which could be of independent interest. We then apply this general fact by establishing matrix Poincaré inequalities to derive matrix concentration inequalities for Gaussian measures, product measures and for Strong…
We use a generalization of Hoeffding's inequality to show concentration results for the free energy of disordered pinning models, assuming only that the disorder has a finite exponential moment. We also prove some concentration inequalities for directed polymers in random environment, which we use to establish a large deviations results for the end position of the polymer under the polymer measure.
We provide a new exponential concentration inequality for First Passage Percolation valid for a wide class of edge times distributions. This improves and extends a result by Benjamini, Kalai and Schramm which gave a variance bound for Bernoulli edge times. Our approach is based on some functional inequalities extending the work of Rossignol and Falik and Samorodnitsky.
We prove that an irreducible aperiodic Markov chain is geometrically ergodic if and only if any separately bounded functional of the stationary chain satisfies an appropriate subgaussian deviation inequality from its mean.
The aim of this paper is to prove an improved version of the bounded differences inequality for matrix valued functions, by developing the methods of Mackey et al.: "Matrix Concentration Inequalities via the Method of Exchangeable Pairs". Along the way, we prove new trace inequalities for the matrix exponential.
Using the method of transportation-information inequality introduced in \cite{GLWY}, we establish Bernstein type's concentration inequalities for empirical means $\frac 1t \int_0^t g(X_s)ds$ where $g$ is a unbounded observable of the symmetric Markov process $(X_t)$. Three approaches are proposed : functional inequalities approach ; Lyapunov function method ; and an approach through the Lipschitzian norm of the solution to the Poisson equation. Several applications and examples are studied.
We prove that an approximated version of the Brunn--Minkowski inequality with volume distortion coefficient implies a Gaussian concentration-of-measure phenomenon. Our main theorem is applicable to discrete spaces.
One way to define the concentration of measure phenomenon is via Talagrand inequalities, also called transportation-information inequalities. That is, a comparison of the Wasserstein distance from the given measure to any other absolutely continuous measure with finite relative entropy. Such transportation-information inequalities were recently established for some stochastic differential equations. Here, we develop a similar theory for some stochastic partial differential equations.
This work shows how exponential concentration inequalities for additive functionals of stochastic processes over a finite time interval can be derived from concentration inequalities for martingales. The approach is entirely probabilistic and naturally includes time-inhomogeneous and non-stationary processes as well as initial laws concentrated on a single point. The class of processes studied includes martingales, Markov processes and general square integrable processes. The general approach is complemented by a simple and direct method for martingales, diffusions and discrete-time Markov pro…
Concentration properties of functionals of general Poisson processes are studied. Using a modified $Φ$-Sobolev inequality a recursion scheme for moments is established, which is of independent interest. This is applied to derive moment and concentration inequalities for functionals on abstract Poisson spaces. Applications of the general results in stochastic geometry, namely Poisson cylinder models and Poisson random polytopes, are presented as well.
We prove concentration inequalities for functions of independent random variables {under} sub-gaussian and sub-exponential conditions. The utility of the inequalities is demonstrated by an extension of the now classical method of Rademacher complexities to Lipschitz function classes and unbounded sub-exponential distribution.
We obtain large and moderate deviation estimates, as well as concentration inequalities, for a class of nonuniformly expanding maps with stretched exponential decay of correlations. In the large deviation regime, we also exhibit examples showing that the obtained upper bounds are essentially optimal.
We obtain moderate deviations theorems and exponential (Bernstein type) concentration inequalities for "nonconventional" sums of the form $S_N=\sum_{n=1}^N (F(ξ_{q_1(n)},ξ_{q_2(n)},...,ξ_{q_\ell(n)})-\bar F)$.
For a stochastic process with state space some Polish space, this paper gives sufficient conditions on the initial and conditional distributions for the joint law to satisfy Gaussian concentration inequalities, transportation inequalities and also logarithmic Sobolev inequalities in the case of the Euclidean space. In several cases, the obtained constants are of optimal order of growth with respect to the number of variables, or are independent of this number. These results extend results known for mutually independent variables to weakly dependent variables under Dobrushin-Shlosman type condi…
We prove logarithmic Sobolev inequalities and concentration results for convex functions and a class of product random vectors. The results are used to derive tail and moment inequalities for chaos variables (in spirit of Talagrand and Arcones, Gine). We also show that the same proof may be used for chaoses generated by log-concave random variables, recovering results by Lochowski and present an application to exponential integrability of Rademacher chaos.
This paper deduces exponential matrix concentration from a Poincaré inequality via a short, conceptual argument. Among other examples, this theory applies to matrix-valued functions of a uniformly log-concave random vector. The proof relies on the subadditivity of Poincaré inequalities and a chain rule inequality for the trace of the matrix Dirichlet form. It also uses a symmetrization technique to avoid difficulties associated with a direct extension of the classic scalar argument.
