The purpose of this letter is to improve Hoeffding's lemma and consequently Hoeffding's tail bounds. The improvement pertains to left skewed zero mean random variables $X\in[a,b]$, where $a<0$ and $-a>b$. The proof of Hoeffding's improved lemma uses Taylor's expansion, the convexity of $\exp(sx), s\in {\bf R}$ and an unnoticed observation since Hoeffding's publication in 1963 that for $-a>b$ the maximum of the intermediate function $τ(1-τ)$ appearing in Hoeffding's proof is attained
at an endpoint rather than at $τ=0.5$ as in the case $b>-a$. Using Hoeffding's improved lemma we obtain one sided and two sided tail bounds for $P(S_n\ge t)$ and $P(|S_n|\ge t)$, respectively, where
$S_n=\sum_{i=1}^nX_i$ and the $X_i\in[a_i,b_i],i=1,...,n$ are independent zero mean random variables (not necessarily identically distributed). It is interesting to note that we could also improve Hoeffding's two sided bound for all $\{X_i: a_i\ne b_i,i=1,...,n\}$. This is so because here the one sided bound should be increased by $P(-S_n\ge t)$, wherein the left skewed intervals become right skewed and vice versa.
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
It is well known that Hoeffding's inequality has a lot of applications in the signal and information processing fields. How to improve Hoeffding's inequality and find the refinements of its applications have always attracted much attentions. An improvement of Hoeffding inequality was recently given by Hertz \cite{r1}. Eventhough such an improvement is not so big, it still can be used to update many known results with original Hoeffding's inequality, especially for Hoeffding-Azuma inequality for martingales. However, the results in original Hoeffding's inequality and its refinement one by Hertz…
This paper develops an optimal Chernoff type bound for the probabilities of large deviations of sums $\sum_{k=1}^n f (X_k)$ where $f$ is a real-valued function and $(X_k)_{k \in \mathbb{Z}_{\ge 0}}$ is a finite state Markov chain with an arbitrary initial distribution and an irreducible transition probability matrix satisfying a mild assumption on its positivity pattern, related to the function $f$ being considered. The novelty lies in this being a non-asymptotic finite sample bound. Further, our bound is optimal in the large deviations sense, attaining a constant prefactor and an exponential …
Let $M_n= \fsu X1n$ be a sum of independent random variables such that $ X_k\leq 1$, $\E X_k =0$ and $\E X_k^2=\s_k^2$ for all $k$. Hoeffding 1963, Theorem 3, proved that $$¶{M_n \geq nt}\leq H^n(t,p),\quad H(t,p)= \bgl(1+qt/p\bgr)^{p +qt} \bgl({1-t}\bgr)^{q -qt}$$ with $$q=\ffrac 1{1+\s^2},\quad p=1-q, \quad \s^2 =\ffrac {\s_1^2+...+\s_n^2}n,\quad 0<t<1.$$ Bentkus 2004 improved Hoeffding's inequalities using binomial tails as upper bounds. Let $\ga_k =\E X_k^3/\s_k^3$ and $ \vk_k= \E X_k^4/\s_k^4$ stand for the skewness and kurtosis of $X_k$. In this paper we prove (improved) counterparts of …
Hoeffding has shown that tail bounds on the distribution for sampling from a finite population with replacement also apply to the corresponding cases of sampling without replacement. (A special case of this result is that binomial tail bounds apply to the corresponding hypergeometric tails.) We give a new proof of Hoeffding's result by constructing a martingale coupling between the sampling distributions. This construction is given by an explicit combinatorial procedure involving balls and urns. We then apply this construction to create martingale couplings between other pairs of sampling dist…
We investigate the properties of a discrete-time martingale $\{X_m\}_{m\in \mathbb Z_{\geq 0}}$, where all differences between adjacent random variables are limited to be not more than a constant as a promise. In this situation, it is known that the Azuma-Hoeffding inequality holds, which gives an upper bound of a probability for exceptional events. The inequality gives a simple form of the upper bound, and it has been utilized for many investigations. However, the inequality is not tight. We give an explicit expression of a tight upper bound, and we show that it and the bound obtained from th…
The Hoeffding-type-inequalities are obtained for the distribution tails of canonical (degenerate) U- and V-statistics of an arbitrary order based on samples from a stationary sequence of observations satisfying $ρ$-mixing.
Suppose that $(Z_n)_{n\geq0}$ is a supercritical branching process in independent and identically distributed random environment. The right tail function of the scaled growth rate for $(Z_n)_{n\geq0}$ is studied. The upper bounds for $\displaystyle\mathbb{P}\left[\frac{\log Z_n}{Mn}-μ\geq x\right]$ for any $x\geq3$ are obtained, by applying an extension of the Hoeffding type inequalities.
Hoeffding-type exponential inequalities are obtained for the distribution tails of canonical von Mises' statistics of arbitrary order based on samples from a stationary sequence of random variables satisfying the φ-mixing condition. The present paper weakens the restrictions on the coefficient φ() which are contained in the paper "Exponential inequalities for the distributions of canonical U- and V-statistics of dependent observations" by Borisov and Volodko (2009). At the same time, we correct the corresponding proof in this paper.
This work prepares new probability bounds for sums of random, independent, Hermitian tensors. These probability bounds characterize large-deviation behavior of the extreme eigenvalue of the sums of random tensors. We extend Lapalace transform method and Lieb's concavity theorem from matrices to tensors, and apply these tools to generalize the classical bounds associated with the names Chernoff, Bennett, and Bernstein from the scalar to the tensor setting. Tail bounds for the norm of a sum of random rectangular tensors are also derived from corollaries of random Hermitian tensors cases. The pro…
We consider the problem of finding the optimal upper bound for the tail probability of a sum of $k$ nonnegative, independent and identically distributed random variables with given mean $x$. For $k=1$ the answer is given by Markov's inequality and for $k=2$ the solution was found by Hoeffding and Shrikhande in 1955. We solve the problem for $k=3$ as well as for general $k$ and $x\leq1/(2k-1)$ by showing that it follows from the fractional version of an extremal graph theory problem of Erdős on matchings in hypergraphs.
This paper is the Part II of a serious work about T product tensors focusing at establishing new probability bounds for sums of random, independent, T product tensors. These probability bounds characterize large deviation behavior of the extreme eigenvalue of the sums of random T product tensors. We apply Lapalace transform method and Lieb concavity theorem for T product tensors obtained from our Part I paper, and apply these tools to generalize the classical bounds associated with the names Chernoff, and Bernstein from the scalar to the T product tensor setting. Tail bounds for the norm of a …
We consider a random variable X satisfying almost-sure conditions involving G:=<DX,-DL^{-1}X> where DX is X's Malliavin derivative and L^{-1} is the inverse Ornstein-Uhlenbeck operator. A lower- (resp. upper-) bound condition on G is proved to imply a Gaussian-type lower (resp. upper) bound on the tail P[X>z]. Bounds of other natures are also given. A key ingredient is the use of Stein's lemma, including the explicit form of the solution of Stein's equation relative to the function 1_{x>z}, and its relation to G. Another set of comparable results is established, without the use of Stein's lemm…
This paper presents new probability inequalities for sums of independent, random, self-adjoint matrices. These results place simple and easily verifiable hypotheses on the summands, and they deliver strong conclusions about the large-deviation behavior of the maximum eigenvalue of the sum. Tail bounds for the norm of a sum of random rectangular matrices follow as an immediate corollary. The proof techniques also yield some information about matrix-valued martingales. In other words, this paper provides noncommutative generalizations of the classical bounds associated with the names Azuma, Benn…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Duke University（杜克大学）
The non-asymptotic tail bounds of random variables play crucial roles in probability, statistics, and machine learning. Despite much success in developing upper bounds on tail probability in literature, the lower bounds on tail probabilities are relatively fewer. In this paper, we introduce systematic and user-friendly schemes for developing non-asymptotic lower bounds of tail probabilities. In addition, we develop sharp lower tail bounds for the sum of independent sub-Gaussian and sub-exponential random variables, which match the classic Hoeffding-type and Bernstein-type concentration inequal…
We show sharp bounds for probabilities of large deviations for sums of independent random variables satisfying Bernstein's condition. One such bound is very close to the tail of the standard Gaussian law in certain case; other bounds improve the inequalities of Bennett and Hoeffding by adding missing factors in the spirit of Talagrand (1995). We also complete Talagrand's inequality by giving a lower bound of the same form, leading to an equality. As a consequence, we obtain large deviation expansions similar to those of Cramér (1938), Bahadur-Rao (1960) and Sakhanenko (1991). We also show that…
Under the assumption that the distribution of a nonnegative random variable $X$ admits a bounded coupling with its size biased version, we prove simple and strong concentration bounds. In particular the upper tail probability is shown to decay at least as fast as the reciprocal of a Gamma function, guaranteeing a moment generating function that converges everywhere. The class of infinitely divisible distributions with finite mean, whose Lévy measure is supported on an interval contained in $[0,c]$ for some $c < \infty$, forms a special case in which this upper bound is logarithmically sharp. I…
We re-examine a lower-tail upper bound for the random variable $$X=\prod_{i=1}^{\infty}\min\left\{\sum_{k=1}^iE_k,1\right\},$$ where $E_1,E_2,\ldots\stackrel{iid}\sim\text{Exp}(1)$. This bound has found use in root-finding and seed-finding algorithms for randomly growing trees, and was initially proved as a lemma in the context of the uniform attachment tree model. We first show that $X$ has a useful representation as a compound product of uniform random variables that allows us to determine its moments and refine the existing nonasymptotic bound. Next we demonstrate that the lower-tail probab…
For any fixed simple graph $H=(V,E)$ and any fixed $u>0$, we establish the leading order of the exponential rate function for the probability that the number of copies of $H$ in the Erdős--Rényi graph $G(n,p)$ exceeds its expectation by a factor $1+u$, assuming $n^{-κ(H)}\ll p\ll1$, with $κ(H) = 1/(2Δ)$, where $Δ\ge 1$ is the maximum degree of $H$. This improves on a previous result of Chatterjee and the second author, who obtained $κ(H)=c/(Δ|E|)$ for a constant $c>0$. Moreover, for the case of cycle counts we can take $κ$ as large as $1/2$. We additionally obtain the sharp upper tail for Scha…
We investigate links between the so-called Stein's density approach in dimension one and some functional and concentration inequalities. We show that measures having a finite first moment and a density with connected support satisfy a weighted Poincaré inequality with the weight being the Stein kernel, that indeed exists and is unique in this case. Furthermore, we prove weighted log-Sobolev and asymmetric Brascamp-Lieb type inequalities related to Stein kernels. We also show that existence of a uniformly bounded Stein kernel is sufficient to ensure a positive Cheeger isoperimetric constant. Th…
Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（北京未来区块链与隐私计算高精尖创新中心）；Advanced Institute of Information Technology, Peking University（北京大学信息技术高等研究院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机技术研究所）；Computer and Mathematical Sciences, Computer Science, and Statistics, University of Toronto（多伦多大学计算机与数学科学、计算机科学和统计学系）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
