The well-known Bennett-Hoeffding bound for sums of independent random variables is refined, by taking into account truncated third moments, and at that also improved by using, instead of the class of all increasing exponential functions, the much larger class of all generalized moment functions f such that f and f" are increasing and convex. It is shown that the resulting bounds have certain optimality properties. Comparisons with related known bounds are given. The results can be extended in a standard manner to (the maximal functions of) (super)martingales.
We prove a Bennett-type concentration bound for suprema of empirical processes based on sampling without replacement and a corresponding bound in the case of an arbitrary Hoeffding statistics. We improve on the previous results of such type, providing a sharper concentration profile.
We give an extension of Hoeffding's inequality to the case of supermartingales with differences bounded from above. Our inequality strengthens or extends the inequalities of Freedman, Bernstein, Prohorov, Bennett and Nagaev.
This note makes the obvious observation that Hoeffding's original proof of his inequality remains valid in the game-theoretic framework. All details are spelled out for the convenience of future reference.
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
It is well known that Hoeffding's inequality has a lot of applications in the signal and information processing fields. How to improve Hoeffding's inequality and find the refinements of its applications have always attracted much attentions. An improvement of Hoeffding inequality was recently given by Hertz \cite{r1}. Eventhough such an improvement is not so big, it still can be used to update many known results with original Hoeffding's inequality, especially for Hoeffding-Azuma inequality for martingales. However, the results in original Hoeffding's inequality and its refinement one by Hertz…
We prove a multivariate version of Hoeffding's inequality about the distribution of homogeneous polynomials of Rademacher functions. The proof is based on such an estimate about the moments of homogeneous polynomials of Rademacher functions which can be considered as an improvement of Borell's inequality in a most important special case.
In this article, we establish Hoeffding's inequality for bounded Lipschitz functions of a class of not necessarily irreducible Markov models. The result complements the existing literature on this topic where Hoeffding's inequality for bounded measurable functions of a class of irreducible Markov models has been considered. Our approach is based on the assumption of uniform ergodicity of the underlying Markov model in $L^1$-Wasserstein space.
In this note, we present a version of Hoeffding's inequality in a continuous-time setting, where the data stream comes from a uniformly ergodic diffusion process. Similar to the well-studied case of Hoeffding's inequality for discrete-time uniformly ergodic Markov chain, the proof relies on techniques ranging from martingale theory to classical Hoeffding's lemma as well as the notion of deviation kernel of diffusion process. We present two examples to illustrate our results. In the first example we consider large deviation probability on the occupation time of the Jacobi diffusion, a popular p…
We show that the Bernstein-Hoeffding method can be employed to a larger class of generalized moments. This class includes the exponential moments whose properties play a key role in the proof of a well-known inequality of Wassily Hoeffding, for sums of independent and bounded random variables whose mean is assumed to be known. As a result we can generalise and improve upon this inequality. We show that Hoeffding's bound is optimal in a broader sense. Our approach allows to obtain "missing" factors in Hoeffding's inequality whose existence is motivated by the central limit theorem. The later re…
We prove deviation bounds for the random variable $\sum_{i=1}^{n} f_i(Y_i)$ in which $\{Y_i\}_{i=1}^{\infty}$ is a Markov chain with stationary distribution and state space $[N]$, and $f_i: [N] \rightarrow [-a_i, a_i]$. Our bound improves upon previously known bounds in that the dependence is on $\sqrt{a_1^2+\cdots+a_n^2}$ rather than $\max_{i}\{a_i\}\sqrt{n}.$ We also prove deviation bounds for certain types of sums of vector--valued random variables obtained from a Markov chain in a similar manner. One application includes bounding the expected value of the Schatten $\infty$-norm of a random…
This paper develops an optimal Chernoff type bound for the probabilities of large deviations of sums $\sum_{k=1}^n f (X_k)$ where $f$ is a real-valued function and $(X_k)_{k \in \mathbb{Z}_{\ge 0}}$ is a finite state Markov chain with an arbitrary initial distribution and an irreducible transition probability matrix satisfying a mild assumption on its positivity pattern, related to the function $f$ being considered. The novelty lies in this being a non-asymptotic finite sample bound. Further, our bound is optimal in the large deviations sense, attaining a constant prefactor and an exponential …
We provide a systematic approach to deal with the following problem. Let $X_1,\ldots,X_n$ be, possibly dependent, $[0,1]$-valued random variables. What is a sharp upper bound on the probability that their sum is significantly larger than their mean? In the case of independent random variables, a fundamental tool for bounding such probabilities is devised by Wassily Hoeffding. In this paper we consider analogues of Hoeffding's result for sums of dependent random variables for which we have certain information on their dependency structure. We prove a result that yields concentration inequalitie…
In this paper we extend the Bernstein, Prohorov and Bennett inequalities to the noncommutative setting. In addition we provide an improved version of the noncommutative Rosenthal inequality, essentially due to Nagaev, Pinelis and Pinelis, Utev for commutative random variables. We also present new best constants in Rosenthal's inequality. Applying these results to random Fourier projections, we recover and elaborate on fundamental results from compressed sensing, due to Candes, Romberg and Tao.
Let $S_N$ be the sum of vector-valued functions defined on a finite Markov chain. An analogue of the Bernstein--Hoeffding inequality is derived for the probability of large deviations of $S_N$ and relates the probability to the spectral gap of the Markov chain. Examples suggest that this inequality is better than alternative inequalities if the chain has a sufficiently large spectral gap and the function is high-dimensional.
We investigate the properties of a discrete-time martingale $\{X_m\}_{m\in \mathbb Z_{\geq 0}}$, where all differences between adjacent random variables are limited to be not more than a constant as a promise. In this situation, it is known that the Azuma-Hoeffding inequality holds, which gives an upper bound of a probability for exceptional events. The inequality gives a simple form of the upper bound, and it has been utilized for many investigations. However, the inequality is not tight. We give an explicit expression of a tight upper bound, and we show that it and the bound obtained from th…
In a celebrated work by Hoeffding [J. Amer. Statist. Assoc. 58 (1963) 13-30], several inequalities for tail probabilities of sums M_n=X_1+... +X_n of bounded independent random variables X_j were proved. These inequalities had a considerable impact on the development of probability and statistics, and remained unimproved until 1995 when Talagrand [Inst. Hautes Etudes Sci. Publ. Math. 81 (1995a) 73-205] inserted certain missing factors in the bounds of two theorems. By similar factors, a third theorem was refined by Pinelis [Progress in Probability 43 (1998) 257-314] and refined (and extended) …
We obtain a Bernstein type Gaussian concentration inequality for martingales. Our inequality improves the Azuma-Hoeffding inequality for moderate deviations $x$. Following the work of McDiarmid (1989), Talagrand (1996) and Boucheron, Lugosi and Massart (2000,2003), we show that our result can be applied to the concentration of random functions, Erdös-Rényi random graph, and maxima of empirical processes. Several interesting Gaussian concentration inequalities have been obtained.
We prove a new Bernstein-type inequality for the log-likelihood function of Bernoulli variables. In contrast to classical Bernstein's inequality and Hoeffding's inequality when applied to the log-likelihood, the new bound is independent of the parameters of the Bernoulli variables and therefore does not blow up as the parameters approach 0 or 1. The new inequality strengthens certain theoretical results on likelihood-based methods for community detection in networks and can be applied to other likelihood-based methods for binary data.
We show sharp bounds for probabilities of large deviations for sums of independent random variables satisfying Bernstein's condition. One such bound is very close to the tail of the standard Gaussian law in certain case; other bounds improve the inequalities of Bennett and Hoeffding by adding missing factors in the spirit of Talagrand (1995). We also complete Talagrand's inequality by giving a lower bound of the same form, leading to an equality. As a consequence, we obtain large deviation expansions similar to those of Cramér (1938), Bahadur-Rao (1960) and Sakhanenko (1991). We also show that…
For every given real value of the ratio $μ:=A_X/G_X>1$ of the arithmetic and geometric means of a positive random variable $X$ and every real $v>0$, exact upper bounds on the right- and left-tail probabilities $\mathsf{P}(X/G_X\ge v)$ and $\mathsf{P}(X/G_X\le v)$ are obtained, in terms of $μ$ and $v$. In particular, these bounds imply that $X/G_X\to1$ in probability as $A_X/G_X\downarrow1$. Such a result may be viewed as a converse to a reverse Jensen inequality for the strictly concave function $f=\ln$, whereas the well-known Cantelli and Chebyshev inequalities may be viewed as converses to a…
The works of Bennett, Carbery, Christ, Tao and of Valdimarsson have clarified when equality holds in the Brascamp-Lieb inequality. Here we characterize the case of equality in the Geometric case of Barthe's reverse Brascamp-Lieb inequality.
We establish concentration inequalities for Lipschitz functions of dependent random variables, whose dependencies are specified by forests. We also give concentration results for decomposable functions, improving Janson's Hoeffding-type inequality for the summation of graph-dependent bounded variables. These results extend McDiarmid's bounded difference inequality to the dependent cases.
A new characterization of the exponential type Orlicz spaces generated by the functions $\exp(|x|^p)-1$ ($p\ge 1$) is given. We define norms for centered random variables belonging to these spaces. We show equivalence of these norms with the Luxemburg norms. On the example of the Hoeffding inequality we present some application of these norms in a probabilistic context.
We use a generalization of Hoeffding's inequality to show concentration results for the free energy of disordered pinning models, assuming only that the disorder has a finite exponential moment. We also prove some concentration inequalities for directed polymers in random environment, which we use to establish a large deviations results for the end position of the polymer under the polymer measure.
Let $M_n= \fsu X1n$ be a sum of independent random variables such that $ X_k\leq 1$, $\E X_k =0$ and $\E X_k^2=\s_k^2$ for all $k$. Hoeffding 1963, Theorem 3, proved that $$¶{M_n \geq nt}\leq H^n(t,p),\quad H(t,p)= \bgl(1+qt/p\bgr)^{p +qt} \bgl({1-t}\bgr)^{q -qt}$$ with $$q=\ffrac 1{1+\s^2},\quad p=1-q, \quad \s^2 =\ffrac {\s_1^2+...+\s_n^2}n,\quad 0<t<1.$$ Bentkus 2004 improved Hoeffding's inequalities using binomial tails as upper bounds. Let $\ga_k =\E X_k^3/\s_k^3$ and $ \vk_k= \E X_k^4/\s_k^4$ stand for the skewness and kurtosis of $X_k$. In this paper we prove (improved) counterparts of …
The Biodesign Institute（生物设计研究所）；Nanjing University of Science and Technology（南京理工大学）；Arizona State University（亚利桑那州立大学）
In this paper, we provide a new framework to obtain the generalization bounds of the learning process for domain adaptation, and then apply the derived bounds to analyze the asymptotical convergence of the learning process. Without loss of generality, we consider two kinds of representative domain adaptation: one is with multiple sources and the other is combining source and target data. In particular, we use the integral probability metric to measure the difference between two domains. For either kind of domain adaptation, we develop a related Hoeffding-type deviation inequality and a symmetr…
Concentration inequalities are fundamental tools in probabilistic combinatorics and theoretical computer science for proving that random functions are near their means. Of particular importance is the case where f(X) is a function of independent random variables X=(X_1, ..., X_n). Here the well known bounded differences inequality (also called McDiarmid's or Hoeffding-Azuma inequality) establishes sharp concentration if the function f does not depend too much on any of the variables. One attractive feature is that it relies on a very simple Lipschitz condition (L): it suffices to show that |f(…
