It is well-known that each statistic in the family of power divergence statistics, across $n$ trials and $r$ classifications with index parameter $λ\in\mathbb{R}$ (the Pearson, likelihood ratio and Freeman-Tukey statistics correspond to $λ=1,0,-1/2$, respectively) is asymptotically chi-square distributed as the sample size tends to infinity. In this paper, we obtain explicit bounds on this distributional approximation, measured using smooth test functions, that hold for a given finite sample $n$, and all index parameters ($λ>-1$) for which such finite sample bounds are meaningful. We obtain bounds that are of the optimal order $n^{-1}$. The dependence of our bounds on the index parameter $λ$ and the cell classification probabilities is also optimal, and the dependence on the number of cells is also respectable. Our bounds generalise, complement and improve on recent results from the literature.
We propose a new definition of the chi-square divergence between distributions. Based on convexity properties and duality, this version of the χ^2 is well suited both for the classical applications of the χ^2 for the analysis of contingency tables and for the statistical tests for parametric models, for which it has been advocated to be robust against inliers. We present two applications in testing. In the first one we deal with tests for finite and infinite numbers of linear constraints, while, in the second one, we apply χ^2-methodology for parametric testing against contamination.
We introduce a new test procedure of independence in the framework of parametric copulas with unknown marginals. The method is based essentially on the dual representation of $χ^2$-divergence on signed finite measures. The asymptotic properties of the proposed estimate and the test statistic are studied under the null and alternative hypotheses, with simple and standard limit distributions both when the parameter is an interior point or not.
For testing goodness of fit it is very popular to use either the chi square statistic or G statistics (information divergence). Asymptotically both are chi square distributed so an obvious question is which of the two statistics that has a distribution that is closest to the chi square distribution. Surprisingly, when there is only one degree of freedom it seems like the distribution of information divergence is much better approximated by a chi square distribution than the chi square statistic. For random variables we introduce a new transformation that transform several important distributio…
The Wald test statistic has been shown to diverge (Dufour et al, 2013, 2017) under some conditions. This note links the divergence to eigenvalues of a polynomial matrix and establishes the divergence rate.
Stein Variational Gradient Descent (SVGD), a popular sampling algorithm, is often described as the kernelized gradient flow for the Kullback-Leibler divergence in the geometry of optimal transport. We introduce a new perspective on SVGD that instead views SVGD as the (kernelized) gradient flow of the chi-squared divergence which, we show, exhibits a strong form of uniform exponential ergodicity under conditions as weak as a Poincaré inequality. This perspective leads us to propose an alternative to SVGD, called Laplacian Adjusted Wasserstein Gradient Descent (LAWGD), that can be implemented fr…
In this paper we apply divergence measures to empirical likelihood applied to logistic regression models. We define a family of empirical test statistics based on divergence measures, called empirical phi-divergence test statistics, extending the empirical likelihood ratio test. We study the asymptotic distribution of these empirical test statistics, showing that it is the same for all the test statistics in this family, and the same as the classical empirical likelihood ratio test. Next, we study the power function for the members in this family, showing that the empirical phi-divergence test…
Statistical inference based on divergence measures have a long history. Recently, Maji, Ghosh and Basu (2014) have introduced a general family of divergences called the logarithmic super divergence (LSD) family. This family acts as a superfamily for both of the logarithmic power divergence (LPD) family (eg. Renyi, 1961) and the logarithmic density power divergence (LDPD)family introduced by Jones et al. (2001). In this paper we describe the asymptotic properties of the inference procedures resulting from this divergence in discrete models. The properties are well supported by real data example…
In this paper we propose the use of $ϕ$-divergences as test statistics to verify simple hypotheses about a one-dimensional parametric diffusion process $\de X_t = b(X_t, θ)\de t + σ(X_t, θ)\de W_t$, from discrete observations $\{X_{t_i}, i=0, ..., n\}$ with $t_i = iΔ_n$, $i=0, 1, >..., n$, under the asymptotic scheme $Δ_n\to0$, $nΔ_n\to\infty$ and $nΔ_n^2\to 0$. The class of $ϕ$-divergences is wide and includes several special members like Kullback-Leibler, Rényi, power and $α$-divergences. We derive the asymptotic distribution of the test statistics based on $ϕ$-divergences. The limiting law …
We investigate optimal posteriors for recently introduced \cite{begin2016pac} chi-squared divergence based PAC-Bayesian bounds in terms of nature of their distribution, scalability of computations, and test set performance. For a finite classifier set, we deduce bounds for three distance functions: KL-divergence, linear and squared distances. Optimal posterior weights are proportional to deviations of empirical risks, usually with subset support. For uniform prior, it is sufficient to search among posteriors on classifier subsets ordered by these risks. We show the bound minimization for linea…
It is proved that the information divergence statistic is infinitely more Bahadur efficient than the power divergence statistics of the orders $α>1$ as long as the sequence of alternatives is contiguous with respect to the sequence of null-hypotheses and the the number of observations per bin increases to infinity is not very slow. This improves the former result in Harremoës and Vajda (2008) where the the sequence of null-hypotheses was assumed to be uniform and the restrictions on on the numbers of observations per bin were sharper. Moreover, this paper evaluates also the Bahadur efficiency …
Recently Batsidis \textit{et al.} (2011) have presented a new procedure based on divergence measures for testing the hypothesis of the existence of a change point in exponential populations. A simulation study was carried out, in this paper, using the asymptotic critical points obtained from the asymptotic distribution of the new test statistics introduced there. The main purpose of this paper is to study the behavior of the test statistics introduced in the cited paper of Batsidis \textit{et al.} (2011), using simulated critical points.
This paper studies the change point problem for a general parametric, univariate or multivariate family of distributions. An information theoretic procedure is developed which is based on general divergence measures for testing the hypothesis of the existence of a change. For comparing the accuracy of the new test-statistic a simulation study is performed for the special case of a univariate discrete model. Finally, the procedure proposed in this paper is illustrated through a classical change-point example.
Minimum divergence methods are popular tools in a variety of statistical applications. We consider tubular model adequacy tests, and demonstrate that the new divergences that are generated in the process are very useful in robust statistical inference. In particular we show that the family of $S$-divergences can be alternatively developed using the tubular model adequacy tests; a further application of the paradigm generates a larger superfamily of divergences. We describe the properties of this larger class and its potential applications in robust inference. Along the way, the failure of the …
Projection theorems of divergences enable us to find reverse projection of a divergence on a specific statistical model as a forward projection of the divergence on a different but rather "simpler" statistical model, which, in turn, results in solving a system of linear equations. Reverse projection of divergences are closely related to various estimation methods such as the maximum likelihood estimation or its variants in robust statistics. We consider projection theorems of three parametric families of divergences that are widely used in robust statistics, namely the Rényi divergences (or th…
Statistical divergences (SDs), which quantify the dissimilarity between probability distributions, are a basic constituent of statistical inference and machine learning. A modern method for estimating those divergences relies on parametrizing an empirical variational form by a neural network (NN) and optimizing over parameter space. Such neural estimators are abundantly used in practice, but corresponding performance guarantees are partial and call for further exploration. We establish non-asymptotic absolute error bounds for a neural estimator realized by a shallow NN, focusing on four popula…
One of the most common methods for statistical inference is the maximum likelihood estimator (MLE). The MLE needs to compute the normalization constant in statistical models, and it is often intractable. Using unnormalized statistical models and replacing the likelihood with the other scoring rule are a good way to circumvent such high computation cost, where the scoring rule measures the goodness of fit of the model to observed samples. The scoring rule is closely related to the Bregman divergence, which is a discrepancy measure between two probability distributions. In this paper, the purpos…
Divergence measures have a long association with statistical inference, machine learning and information theory. The density power divergence and related measures have produced many useful (and popular) statistical procedures, which provide a good balance between model efficiency on one hand and outlier stability or robustness on the other. The logarithmic density power divergence, a particular logarithmic transform of the density power divergence, has also been very successful in producing efficient and stable inference procedures; in addition it has also led to significant demonstrated appli…
This paper introduces a new superfamily of divergences that is similar in spirit to the S-divergence family introduced by Ghosh et al. (2013). This new family serves as an umbrella that contains the logarithmic power divergence family (Renyi, 1961; Maji, Chakraborty and Basu 2014) and the logarithmic density power divergence family (Jones et al., 2001) as special cases. Various properties of this new family and the corresponding minimum distance procedures are discussed with particular emphasis on the robustness issue; these properties are demonstrated through simulation studies. In particular…
In this paper, we derive a useful lower bound for the Kullback-Leibler divergence (KL-divergence) based on the Hammersley-Chapman-Robbins bound (HCRB). The HCRB states that the variance of an estimator is bounded from below by the Chi-square divergence and the expectation value of the estimator. By using the relation between the KL-divergence and the Chi-square divergence, we show that the lower bound for the KL-divergence which only depends on the expectation value and the variance of a function we choose. This lower bound can also be derived from an information geometric approach. Furthermor…
A unified family of goodness-of-fit tests based on $ϕ$-divergences is introduced and studied. The new family of test statistics $S_n(s)$ includes both the supremum version of the Anderson--Darling statistic and the test statistic of Berk and Jones [Z. Wahrsch. Verw. Gebiete 47 (1979) 47--59] as special cases ($s=2$ and $s=1$, resp.). We also introduce integral versions of the new statistics. We show that the asymptotic null distribution theory of Berk and Jones [Z. Wahrsch. Verw. Gebiete 47 (1979) 47--59] and Wellner and Koltchinskii [High Dimensional Probability III (2003) 321--332. Birkhäuse…
In this paper a robust version of the classical Wald test statistics for linear hypothesis in the logistic regression model is introduced and its properties are explored. We study the problem under the assumption of random covariates although some ideas with non random covariates are also considered. The family of tests considered is based on the minimum density power divergence estimator instead of the maximum likelihood estimator and it is referred to as the Wald-type test statistic in the paper. We obtain the asymptotic distribution and also study the robustness properties of the Wald type …
The main purpose of this paper is to introduce first a new family of empirical test statistics for testing a simple null hypothesis when the vector of parameters of interest are defined through a specific set of unbiased estimating functions. This family of test statistics is based on a distance between two probability vectors, with the first probability vector obtained by maximizing the empirical likelihood on the vector of parameters, and the second vector defined from the fixed vector of parameters under the simple null hypothesis. The distance considered for this purpose is the phi-diverge…
Stationary ergodic processes with finite alphabets are estimated by finite memory processes from a sample, an n-length realization of the process, where the memory depth of the estimator process is also estimated from the sample using penalized maximum likelihood (PML). Under some assumptions on the continuity rate and the assumption of non-nullness, a rate of convergence in $\bar{d}$-distance is obtained, with explicit constants. The result requires an analysis of the divergence of PML Markov order estimators for not necessarily finite memory processes. This divergence problem is investigated…
We present a new non-parametric statistic, called the weighed $\ell_2$ divergence, based on empirical distributions for sequential change detection. We start by constructing the weighed $\ell_2$ divergence as a fundamental building block for two-sample tests and change detection. The proposed statistic is proved to attain the optimal sample complexity in the offline setting. We then study the sequential change detection using the weighed $\ell_2$ divergence and characterize the fundamental performance metrics, including the average run length (ARL) and the expected detection delay (EDD). We al…
We introduce estimation and test procedures through divergence minimiza- tion for models satisfying linear constraints with unknown parameter. These procedures extend the empirical likelihood (EL) method and share common features with generalized empirical likelihood approach. We treat the problems of existence and characterization of the divergence projections of probability distributions on sets of signed finite measures. We give a precise characterization of duality, for the proposed class of estimates and test statistics, which is used to derive their limiting distributions (including the …
In this paper, we establish some new central limit theorems for certain spectral statistics of a high-dimensional sample covariance matrix under a divergent spectral norm population model. This model covers the divergent spiked population model as a special case. Meanwhile, the number of the spiked eigenvalues can either be fixed or grow to infinity. It is seen from our theorems that the divergence of population spectral norm affects the fluctuations of the linear spectral statistics in a fickle way, depending on the divergence rate.
The autodependogram is a graphical device recently proposed in the literature to analyze autodependencies. It is defined computing the classical Pearson chi-square statistics of independence at various lags in order to point out the presence lag-depedencies. This paper proposes an improvement of this diagram obtained by substituting the chi-square statistics with an estimator of the Kulback-Leibler divergence between the bivariate density of two delayed variables and the product of their marginal distributions. A simulation study, on well-established time series models, shows that this new aut…
In order to adapt the Wasserstein distance to the large sample multivariate non-parametric two-sample problem, making its application computationally feasible, permutation tests based on the Sinkhorn divergence between probability vectors associated to data dependent partitions are considered. Different ways of implementing these tests are evaluated and the asymptotic distribution of the underlying statistic is established in some cases. The statistics proposed are compared, in simulated examples, with the test of Schilling's, one of the best non-parametric tests available in the literature.
Density-based directed distances -- particularly known as divergences -- between probability distributions are widely used in statistics as well as in the adjacent research fields of information theory, artificial intelligence and machine learning. Prominent examples are the Kullback-Leibler information distance (relative entropy) which e.g. is closely connected to the omnipresent maximum likelihood estimation method, and Pearson's chisquare-distance which e.g. is used for the celebrated chisquare goodness-of-fit test. Another line of statistical inference is built upon distribution-function-b…
Preserving the robustness of the procedure has, at the present time, become almost a default requirement for statistical data analysis. Since efficiency at the model and robustness under misspecification of the model are often in conflict, it is important to choose such inference procedures which provide the best compromise between these two concepts. Some minimum Bregman divergence estimators and related tests of hypothesis seem to be able to do well in this respect, with the procedures based on the density power divergence providing the existing standard. In this paper we propose a new famil…
In statistical analysis, measuring a score of predictive performance is an important task. In many scientific fields, appropriate scores were tailored to tackle the problems at hand. A proper score is a popular tool to obtain statistically consistent forecasts. Furthermore, a mathematical characterization of the proper score was studied. As a result, it was revealed that the proper score corresponds to a Bregman divergence, which is an extension of the squared distance over the set of probability distributions. In the present paper, we introduce composite scores as an extension of the typical …
