We develop a nonparametric extension of the sequential generalized likelihood ratio (GLR) test and corresponding time-uniform confidence sequences for the mean of a univariate distribution. By utilizing a geometric interpretation of the GLR statistic, we derive a simple analytic upper bound on the probability that it exceeds any prespecified boundary; these are intractable to approximate via simulations due to infinite horizon of the tests and the composite nonparametric nulls under consideration. Using time-uniform boundary-crossing inequalities, we carry out a unified nonasymptotic analysis of expected sample sizes of one-sided and open-ended tests over nonparametric classes of distributions (including sub-Gaussian, sub-exponential, sub-gamma, and exponential families). Finally, we present a flexible and practical method to construct time-uniform confidence sequences that are easily tunable to be uniformly close to the pointwise Chernoff bound over any target time interval.
University of California, San Diego（加州大学圣地亚哥分校）；University of California, Berkeley（加州大学伯克利分校）
We propose a new algorithmic framework for sequential hypothesis testing with i.i.d. data, which includes A/B testing, nonparametric two-sample testing, and independence testing as special cases. It is novel in several ways: (a) it takes linear time and constant space to compute on the fly, (b) it has the same power guarantee as a non-sequential version of the test with the same computational constraints up to a small factor, and (c) it accesses only as many samples as are required - its stopping time adapts to the unknown difficulty of the problem. All our test statistics are constructed to b…
We study the law of the iterated logarithm (LIL) for the maximum likelihood estimation of the parameters (as a convex optimization problem) in the generalized linear models with independent or weakly dependent ($ρ$-mixing, $m$-dependent) responses under mild conditions. The LIL is useful to derive the asymptotic bounds for the discrepancy between the empirical process of the log-likelihood function and the true log-likelihood. As the application of the LIL, the strong consistency of some penalized likelihood based model selection criteria can be shown. Under some regularity conditions, the mod…
University of California, San Diego（加利福尼亚大学圣地亚哥分校）
We give concentration bounds for martingales that are uniform over finite times and extend classical Hoeffding and Bernstein inequalities. We also demonstrate our concentration bounds to be optimal with a matching anti-concentration inequality, proved using the same method. Together these constitute a finite-time version of the law of the iterated logarithm, and shed light on the relationship between it and the central limit theorem.
The aim of this paper is to estimate the density f of a random variable X when one has access to independent observations of the sum of K $\ge$ 2 independent copies of X. We provide a constructive estimator based on a suitable definition of the logarithm of the empirical characteristic function.We propose a new strategy for the data driven choice of the cut-off parameter. The adaptive estimator is proven to be minimax-optimal up to some logarithmic loss. A numerical study illustrates the performances of the method. Moreover, we discuss the fact that the definition of the estimator applies in a…
A confidence sequence is a sequence of confidence intervals that is uniformly valid over an unbounded time horizon. Our work develops confidence sequences whose widths go to zero, with nonasymptotic coverage guarantees under nonparametric conditions. We draw connections between the Cramér-Chernoff method for exponential concentration, the law of the iterated logarithm (LIL), and the sequential probability ratio test -- our confidence sequences are time-uniform extensions of the first; provide tight, nonasymptotic characterizations of the second; and generalize the third to nonparametric settin…
Nonparametric tests via kernel embedding of distributions have witnessed a great deal of practical successes in recent years. However, statistical properties of these tests are largely unknown beyond consistency against a fixed alternative. To fill in this void, we study here the asymptotic properties of goodness-of-fit, homogeneity and independence tests using Gaussian kernels, arguably the most popular and successful among such tests. Our results provide theoretical justifications for this common practice by showing that tests using Gaussian kernel with an appropriately chosen scaling parame…
We consider discrete time models for asset prices with a stationary volatility process. We aim at estimating the multivariate density of this process at a set of consecutive time instants. A Fourier type deconvolution kernel density estimator based on the logarithm of the squared process is proposed to estimate the volatility density. Expansions of the bias and bounds on the variance are derived.
We study the Nonparametric Maximum Likelihood Estimator (NPMLE) for estimating Gaussian location mixture densities in $d$-dimensions from independent observations. Unlike usual likelihood-based methods for fitting mixtures, NPMLEs are based on convex optimization. We prove finite sample results on the Hellinger accuracy of every NPMLE. Our results imply, in particular, that every NPMLE achieves near parametric risk (up to logarithmic multiplicative factors) when the true density is a discrete Gaussian mixture without any prior information on the number of mixture components. NPMLEs can natural…
The construction of confidence regions for parameter vectors is a difficult problem in the nonparametric setting, particularly when the sample size is not large. The bootstrap has shown promise in solving this problem, but empirical evidence often indicates that some bootstrap methods have difficulty in maintaining the correct coverage probability, while other methods may be unstable, often resulting in very large confidence regions. One way to improve the performance of a bootstrap confidence region is to restrict the shape of the region in such a way that the error term of an expansion is as…
Following the works of Berthet (1997), we first obtain exact clustering rates in the functional law of the iterated logarithm for the uniform empirical and quantile processes and for their increments. In a second time, we obtain functional Chung-type limit laws for the local empirical process for a class of target functions on the border of the Strassen set.
We study the law of the iterated logarithm (Khinchin (1924), Kolmogorov (1929)) and related strong invariance principles in stochastic geometry. As potential applications, we think of well-known functionals such as functionals defined on the $k$-nearest neighbors graph and important functionals in topological data analysis such as the Euler characteristic and persistent Betti numbers.
In this paper, we establish uniform-in-bandwidth limit laws of the logarithm for nonparametric Inverse Probability of Censoring Weighted (I.P.C.W.) estimators of the multivariate regression function under random censorship. A similar result is deduced for estimators of the conditional distribution function. The uniform-in-bandwidth consistency for estimators of the conditional density and the conditional hazard rate functions are also derived from our main result. Moreover, the logarithm laws we establish are shown to yield almost sure simultaneous asymptotic confidence bands for the functions…
In this note we prove the following law of the iterated logarithm for the Grenander estimator of a monotone decreasing density: If $f(t_0) > 0$, $f'(t_0) < 0$, and $f'$ is continuous in a neighborhood of $t_0$, then \begin{eqnarray*} \limsup_{n\rightarrow \infty} \left ( \frac{n}{2\log \log n} \right )^{1/3} ( \widehat{f}_n (t_0 ) - f(t_0) ) = \left| f(t_0) f'(t_0)/2 \right|^{1/3} 2M \end{eqnarray*} almost surely where $ M \equiv \sup_{g \in {\cal G}} T_g = (3/4)^{1/3}$ and $ T_g \equiv \mbox{argmax}_u \{ g(u) - u^2 \} $; here ${\cal G}$ is the two-sided Strassen limit set on $R$. The proof re…
Concentration inequalities for strong laws and laws of the iterated logarithm
Department of Mathematics, London School of Economics and Political Science（伦敦政治经济学院数学系）；Department of Statistics and The Miller Institute, University of California, Berkeley（加州大学伯克利分校统计系及米勒研究所）
M-estimators are ubiquitous in machine learning and statistical learning theory. They are used both for defining prediction strategies and for evaluating their precision. In this paper, we propose the first non-asymptotic "any-time" deviation bounds for general M-estimators, where "any-time" means that the bound holds with a prescribed probability for every sample size. These bounds are nonasymptotic versions of the law of iterated logarithm. They are established under general assumptions such as Lipschitz continuity of the loss function and (local) curvature of the population risk. These cond…
We prove Chung-type laws of the iterated logarithm for general Lévy processes at zero. In particular, we provide tools to translate small deviation estimates directly into laws of the iterated logarithm. This reveals laws of the iterated logarithm for Lévy processes at small times in many concrete examples. In some cases, exotic norming functions are derived.
