A Matrix Bernstein Inequality for Strong Rayleigh Distributions
Tarun Kathuria
备注：
This paper was submitted to STOC 2020, during the process of which another paper [ABY19] appeared which proves a slightly stronger result. This note is being released in its unedited form
The Entropy method provides a powerful framework for proving scalar concentration inequalities by establishing functional inequalities like Poincare and log-Sobolev inequalities. These inequalities are especially useful for deriving concentration for dependent distributions coming from stationary distributions of Markov chains. In contrast to the scalar case, the study of matrix concentration inequalities is fairly recent and there does not seem to be a general framework for proving matrix concentration inequalities using the Entropy method. In this paper, we initiate the study of Matrix valued functional inequalities and show how matrix valued functional inequalities can be used to establish a kind of iterative variance comparison argument to prove Matrix Bernstein like inequalities. Strong Rayleigh distributions are a class of distributions which satisfy a strong form of negative dependence properties. We show that scalar Poincare inequalities recently established for a flip-swap markov chain considered in the work of Hermon and Salez [HS19] can be lifted to prove matrix valued Poincare inequalities. This can then be combined with the iterative variance argument to prove a version of matrix Bernstein inequality for Lipshitz matrix-valued functions for Strong Rayleigh measures.
We modify the classical Bernstein's inequality for the sums of independent centered random variables (r.v.) in the terms of relative tails or moments. We built also some examples in order to show the exactness of offered results.
In this paper we obtain a Bernstein type inequality for the sum of self-adjoint centered and geometrically absolutely regular random matrices with bounded largest eigenvalue. This inequality can be viewed as an extension to the matrix setting of the Bernstein-type inequality obtained by Merlevède et al. (2009) in the context of real-valued bounded random variables that are geometrically absolutely regular. The proofs rely on decoupling the Laplace transform of a sum on a Cantor-like set of random matrices.
We give a distribution-dependent concentration inequality for functions of independent variables. The result extends Bernstein's inequality from sums to more general functions, whose variation in any argument does not depend too much on the other arguments. Applications sharpen existing bounds for U-statistics and the generalization error of regularized least squares.
In this paper we obtain a Bernstein type inequality for a class of weakly dependent and bounded random variables. The proofs lead to a moderate deviations principle for sums of bounded random variables with exponential decay of the strong mixing coeficients that complements the large deviation result obtained by Bryc and Dembo (1998) under superexponential mixing rates.
In this paper we present a tail inequality for the maximum of partial sums of a weakly dependent sequence of random variables that are not necessarily bounded. The class considered includes geometrically and subgeometrically strongly mixing sequences. The result is then used to derive asymptotic moderate deviations results. Applications include classes of Markov chains, functions of linear processes with absolutely regular innovations and ARCH models
We prove a multivariate version of Bernstein's inequality about the probability that degenerate $U$-statistics take a value larger than some number $u$. This is an improvement of former estimates for the same problem which yields an asymptotically sharp estimate for not too large numbers $u$. This paper also contains an analogous bound about the distribution of multiple Wiener-Ito integrals. Their comparison shows that our results are sharp. The proofs are based on good estimates about high moments of multiple random integrals. They are obtained by means of a diagram formula which enables us t…
Using the renewal approach we prove Bernstein-like inequalities for additive functionals of geometrically ergodic Markov chains, thus obtaining counterparts of inequalities for sums of independent random variables. The coefficient in the sub-Gaussian part of our estimate is the asymptotic variance of the additive functional, i.e. the variance of the limiting Gaussian variable in the Central Limit Theorem for Markov chains. This refines earlier results by R. Adamczak and W. Bednorz, which were obtained under the additional assumption of strong aperiodicity of the chain.
We consider the three dimensional array $\mathcal{A} = \{a_{i,j,k}\}_{1\le i,j,k \le n}$, with $a_{i,j,k} \in [0,1]$, and the two random statistics $T_{1}:= \sum_{i=1}^n \sum_{j=1}^n a_{i,j,σ(i)}$ and $T_{2}:= \sum_{i=1}^{n} a_{i,σ(i),π(i)}$, where $σ$ and $π$ are chosen independently from the set of permutations of $\{1,2,\ldots,n \}.$ These can be viewed as natural three dimensional generalizations of the statistic $T_{3}=\sum_{i=1}^{n} a_{i,σ(i)}$, considered by Hoeffding \cite{Hoe51}. Here we give Bernstein type concentration inequalities for $T_{1}$ and $T_{2}$ by extending the argument f…
We show that the Bernstein-Hoeffding method can be employed to a larger class of generalized moments. This class includes the exponential moments whose properties play a key role in the proof of a well-known inequality of Wassily Hoeffding, for sums of independent and bounded random variables whose mean is assumed to be known. As a result we can generalise and improve upon this inequality. We show that Hoeffding's bound is optimal in a broader sense. Our approach allows to obtain "missing" factors in Hoeffding's inequality whose existence is motivated by the central limit theorem. The later re…
In this article we present a Bernstein inequality for sums of random variables which are defined on a spatial lattice structure. The inequality can be used to derive concentration inequalities. It can be useful to obtain consistency properties for nonparametric estimators of conditional expectation functions.
We establish a Bernstein-type inequality for a class of stochastic processes that include the classical geometrically $ϕ$-mixing processes, Rio's generalization of these processes, as well as many time-discrete dynamical systems. Modulo a logarithmic factor and some constants, our Bernstein-type inequality coincides with the classical Bernstein inequality for i.i.d.~data. We further use this new Bernstein-type inequality to derive an oracle inequality for generic regularized empirical risk minimization algorithms and data generated by such processes. Applying this oracle inequality to support …
We consider the stochastic integrals of multivariate point processes and study their concentration phenomena. In particular, we obtain a Bernstein type of concentration inequality through Doléans-Dade exponential formula and a uniform exponential inequality using a generic chaining argument. As applications, we obtain a upper bound for a sequence of discrete time martingales indexed by a class of functionals, and so derive the rate of convergence for nonparametric maximum likelihood estimators, which is an improvement of earlier work of van de Geer.
This note contains some estimates for tail spaces on Hamming cube. We use analytic paraproduct operator for that purpose. We also show several types of Bernstein--Markov inequalities for Banach space valued functions on Hamming cube. Here the novelty is in getting rid of some irritating logarithms and in proving Bernstein--Markov inequalities for $|\nabla f|_X$ rather than for $Δ^{1/2} f$ for $X$ valued polynomials on Hamming cube.
We introduce a Bernstein-type inequality which serves to uniformly control quadratic forms of gaussian variables. The latter can for example be used to derive sharp model selection criteria for linear estimation in linear regression and linear inverse problems via penalization, and we do not exclude that its scope of application can be made even broader.
We present a new, very short proof of a conjecture by I. Raşa, which is an inequality involving basic Bernstein polynomials and convex functions. It was affirmed positively very recently by J. Mrowiec, T. Rajba and S. Wąsowicz (2017) by the use of stochastic convex orders, as well as by Abel (2017) who simplified their proof. We give a useful sufficient condition for the verification of some stochastic convex order relations, which in the case of binomial distributions are equivalent to the I. Raşa inequality. We give also the corresponding inequalities for other distributions. Our methods all…
We present some extensions of Bernstein's concentration inequality for random matrices. This inequality has become a useful and powerful tool for many problems in statistics, signal processing and theoretical computer science. The main feature of our bounds is that, unlike the majority of previous related results, they do not depend on the dimension $d$ of the ambient space. Instead, the dimension factor is replaced by the "effective rank" associated with the underlying distribution that is bounded from above by $d$. In particular, this makes an extension to the infinite-dimensional setting po…
Using the method of transportation-information inequality introduced in \cite{GLWY}, we establish Bernstein type's concentration inequalities for empirical means $\frac 1t \int_0^t g(X_s)ds$ where $g$ is a unbounded observable of the symmetric Markov process $(X_t)$. Three approaches are proposed : functional inequalities approach ; Lyapunov function method ; and an approach through the Lipschitzian norm of the solution to the Poisson equation. Several applications and examples are studied.
We prove a new Bernstein-type inequality for the log-likelihood function of Bernoulli variables. In contrast to classical Bernstein's inequality and Hoeffding's inequality when applied to the log-likelihood, the new bound is independent of the parameters of the Bernoulli variables and therefore does not blow up as the parameters approach 0 or 1. The new inequality strengthens certain theoretical results on likelihood-based methods for community detection in networks and can be applied to other likelihood-based methods for binary data.
The Bernstein inequality is a tight upper bound on tail probabilities for independent random variables. Freedman extended the Bernstein inequality to martingales with differences bounded from above, and then Dzhaparidze and van Zanten generalized Freedman's result to non-bounded locally square integrable martingales. In this paper, we derive some Dzhaparidze-van Zanten type inequalities for self-normalized martingales with square integrable and non-square integrable differences.
Let X_1 ,..., X_n be a collection of binary valued random variables and let f : {0,1}^n -> R be a Lipschitz function. Under a negative dependence hypothesis known as the {\em strong Rayleigh} condition, we show that f - E f satisfies a concentration inequality generalizing the classical Gaussian concentration inequality for sums of independent Bernoullis: P (S_n - E S_n > a) < exp (-2 a^2 / n). The class of strong Rayleigh measures includes determinantal measures, weighted uniform matroids and exclusion measures; some familiar examples from these classes are generalized negative binomials and …
We obtain a Bernstein type Gaussian concentration inequality for martingales. Our inequality improves the Azuma-Hoeffding inequality for moderate deviations $x$. Following the work of McDiarmid (1989), Talagrand (1996) and Boucheron, Lugosi and Massart (2000,2003), we show that our result can be applied to the concentration of random functions, Erdös-Rényi random graph, and maxima of empirical processes. Several interesting Gaussian concentration inequalities have been obtained.
In the present paper we prove that the probabilities of the Pólya urn distribution (with negative replacement) satisfy a monotonicity property similar to that of the binomial distribution (Pólya urn distribution with no replacement). As a consequence, we show that the random variables with Pólya urn distribution (with negative replacement) are stochastically ordered with respect to the parameter giving the initial distribution of the urn. An equivalent formulation of this result shows that the new Bernstein operator recently introduced in [3] is a monotone operator. The proofs are probabilisti…
We introduce two new concepts designed for the study of empirical processes. First, we introduce a new Orlicz norm which we call the Bernstein-Orlicz norm. This new norm interpolates sub-Gaussian and sub-exponential tail behavior. In particular, we show how this norm can be used to simplify the derivation of deviation inequalities for suprema of collections of random variables. Secondly, we introduce chaining and generic chaining along a tree. These simplify the well-known concepts of chaining and generic chaining. The supremum of the empirical process is then studied as a special case. We sho…
Recent development in high-dimensional statistical inference has necessitated concentration inequalities for a broader range of random variables. We focus on sub-Weibull random variables, which extend sub-Gaussian or sub-exponential random variables to allow heavy-tailed distributions. This paper presents concentration inequalities for independent sub-Weibull random variables with finite Generalized Bernstein-Orlicz norms, providing generalized Bernstein's inequalities and Rosenthal-type moment bounds. The tightness of the proposed bounds is shown through lower bounds of the concentration ineq…
We derive concentration inequalities for empirical means $\frac{1}{t} \int_0^t f(X_s) ds$ where $X_s$ is an irreducible Markov jump process on a finite state space and $f$ some observable. Using a Feynman-Kac semigroup we first derive a general concentration inequality. Then, based on this inequality we derive further concentration inequalities. Hereby we use three different methods; perturbation theory, Poincaré inequalities and information inequalities. We also obtain a Bernstein type concentration inequality.
In this paper we study the regularity of paths in terms of properties of admissible nets. We show the right concentration inequality above the modulus of continuity. Using the approach we prove the Bernstein type inequality for the empirical processes. Therefore we obtain the best form of concentration for processes studied recently by Mendelson and Paouris. Results of this type are of importance in the compressed sensing theory.
Let $\pa{X_{t}}_{t\in T}$ be a family of real-valued centered random variables indexed by a countable set $T$. In the first part of this paper, we establish exponential bounds for the deviation probabilities of the supremum $Z=\sup_{t\in T}X_{t}$ by using the generic chaining device introduced in Talagrand (2005). Compared to concentration-type inequalities, these bounds offer the advantage to hold under weaker conditions on the family $\pa{X_{t}}_{t\in T}$. The second part of the paper is oriented towards statistics. We consider the regression setting $Y=f+\eps$ where $f$ is an unknown vector…
We prove a new concentration inequality for U-statistics of order two for uniformly ergodic Markov chains. Working with bounded and $π$-canonical kernels, we show that we can recover the convergence rate of Arcones and Gin{é} who proved a concentration result for U-statistics of independent random variables and canonical kernels. Our result allows for a dependence of the kernels $h_{i,j}$ with the indexes in the sums, which prevents the use of standard blocking tools. Our proof relies on an inductive analysis where we use martingale techniques, uniform ergodicity, Nummelin splitting and Bernst…
Freedman's inequality is a martingale counterpart to Bernstein's inequality. This result shows that the large-deviation behavior of a martingale is controlled by the predictable quadratic variation and a uniform upper bound for the martingale difference sequence. Oliveira has recently established a natural extension of Freedman's inequality that provides tail bounds for the maximum singular value of a matrix-valued martingale. This note describes a different proof of the matrix Freedman inequality that depends on a deep theorem of Lieb from matrix analysis. This argument delivers sharp constan…
Initially motivated by the study of the non-asymptotic properties of non-parametric tests based on permutation methods, concentration inequalities for uniformly permuted sums have been largely studied in the literature. Recently, Delyon et al. proved a new Bernstein-type concentration inequality based on martingale theory. This work presents a new proof of this inequality based on the fundamental inequalities for random permutations of Talagrand. The idea is to first obtain a rough inequality for the square root of the permuted sum, and then, iterate the previous analysis and plug this first i…
A Bernstein-type exponential inequality for (generalized) canonical U-statistics of order 2 is obtained and the Rosenthal and Hoffmann-Jørgensen inequalities for sums of independent random variables are extended to (generalized) U-statistics of any order whose kernels are either nonnegative or canonical
We establish universal modified log-Sobolev inequalities for reversible Markov chains on the boolean lattice $\{0,1\}^n$, under the only assumption that the invariant law $π$ satisfies a form of negative dependence known as the stochastic covering property. This condition is strictly weaker than the strong Rayleigh property, and is satisfied in particular by all determinantal measures, as well as any product measure over the set of bases of a balanced matroid. In the special case where $π$ is $k-$homogeneous, our results imply the celebrated concentration inequality for Lipschitz functions due…
We obtain the following dimension independent Bernstein-Markov inequality in Gauss space: for each $1\leq p<\infty$ there exists a constant $C_p>0$ such that for any $k\geq 1$ and all polynomials $P$ on $\mathbb{R}^{k}$ we have $$ \| \nabla P\|_{L^{p}(\mathbb{R}^{k}, \mathrm{d}γ_k)} \leq C_p (\mathrm{deg}\, P)^{\frac{1}{2}+\frac{1}π\arctan\left(\frac{|p-2|}{2\sqrt{p-1}}\right)}\|P\|_{L^{p}(\mathbb{R}^{k}, \mathrm{d}γ_k)}, $$ where $\mathrm{d}γ_k$ is the standard Gaussian measure on $\mathbb{R}^{k}$. We also show that under some mild growth assumptions on any function $B \in C^{2}((0,\infty))\c…
In this paper we prove exponential inequalities (also called Bernstein's inequality) for fractional martingales. As an immediate corollary, we will discuss weak law of large numbers for fractional martingales under divergence assumption on the $β-$variation of the fractional martingale. A non trivial example of application of this convergence result is proposed.
