The distance and divergence of the probability measures play a central role in statistics, machine learning, and many other related fields. The Wasserstein distance has received much attention in recent years because of its distinctions from other distances or divergences. Although~computing the Wasserstein distance is costly, entropy-regularized optimal transport was proposed to computationally efficiently approximate the Wasserstein distance. The purpose of this study is to understand the theoretical aspect of entropy-regularized optimal transport. In this paper, we~focus on entropy-regularized optimal transport on multivariate normal distributions and $q$-normal distributions. We~obtain the explicit form of the entropy-regularized optimal transport cost on multivariate normal and $q$-normal distributions; this provides a perspective to understand the effect of entropy regularization, which was previously known only experimentally. Furthermore, we obtain the entropy-regularized Kantorovich estimator for the probability measure that satisfies certain conditions. We also demonstrate how the Wasserstein distance, optimal coupling, geometric structure, and statistical efficiency are affected by entropy regularization in some experiments. In particular, our results about the explicit form of the optimal coupling of the Tsallis entropy-regularized optimal transport on multivariate $q$-normal distributions and the entropy-regularized Kantorovich estimator are novel and will become the first step towards the understanding of a more general setting.
This paper proves the consistency property for the regularized maximum likelihood estimators (MLEs) of multivariate Hawkes processes (MHPs). It also develops an alternating minimization type algorithm (AA-iPALM) to compute the MLEs with guaranteed global convergence to the set of stationary points. The performance of this AA-iPALM algorithm on both synthetic and real-world data shows that AA-iPALM consistently improves over iPALM and PALM. Moreover, AA-iPALM is able to identify the causality structure in rental listings on craigslist and herd behavior in the NASDAQ ITCH dataset.
We consider the tuning parameter selection rules for nuclear norm regularized multivariate linear regression (NMLR) in high-dimensional setting. High-dimensional multivariate linear regression is widely used in statistics and machine learning, and regularization technique is commonly applied to deal with the special structures in high-dimensional data. As we know, how to select the tuning parameter is an essential issue for regularization approach and it directly affects the model estimation performance. To the best of our knowledge, there are no rules about the tuning parameter selection for …
An admissible estimator of the eigenvalues of the variance-covariance matrix is given for multivariate normal distributions with respect to the scale-invariant squared error loss.
We study admissibility of a subclass of generalized Bayes estimators of a multivariate normal vector when the variance is unknown, under scaled quadratic loss. Minimaxity is also established for certain of these estimators.
This paper develops recurrence relations for integrals that relate the density of multivariate extended skew-normal (ESN) distribution, including the well-known skew-normal (SN) distribution introduced by Azzalini and Dalla-Valle (1996) and the popular multivariate normal distribution. These recursions offer a fast computation of arbitrary order product moments of the multivariate truncated extended skew-normal and multivariate folded extended skew-normal distributions with the product moments as a byproduct. In addition to the recurrence approach, we realized that any arbitrary moment of the …
This short note considers the problem of testing the null hypothesis that the mean values of two multivariate normal variables are proportional. We show that the usual likelihood ratio $χ^2$-test is valid non-asymptotically. Our proof relies on expressing the test statistic as the minimum eigenvalue of a Wishart variable and using a representation of its distribution using Legendre polynomials.
The problem of estimating a mean matrix of a multivariate complex normal distribution with an unknown covariance matrix is considered under an invariant loss function. By using complex versions of the Stein identity, the Stein-Haff identity, and calculus on eigenvalues, a formula is obtained for an unbiased estimate of the risk of an invariant class of estimators, from which several minimax shrinkage estimators are constructed.
We show that the orthogonal projection operator onto the range of the adjoint of a linear operator T can be represented as UT, where U is an invertible linear operator. Using this representation we obtain a decomposition of a multivariate Normal random variable Y as the sum of a linear transformation of Y that is independent of TY and an affine transformation of TY. We then use this decomposition to prove that the regular conditional distribution of a multivariate Normal random variable Y given a linear transformation TY is again a multivariate Normal distribution. This result is equivalent to…
The asymptotic normality of the maximum likelihood estimator (MLE) under regularity conditions is a cornerstone of statistical theory. In this paper, we give explicit upper bounds on the distributional distance between the distribution of the MLE of a vector parameter, and the multivariate normal distribution. We work with possibly high-dimensional, independent but not necessarily identically distributed random vectors. In addition, we obtain explicit upper bounds even in cases where the MLE cannot be expressed analytically.
We use the delta method and Stein's method to derive, under regularity conditions, explicit upper bounds for the distributional distance between the distribution of the maximum likelihood estimator (MLE) of a $d$-dimensional parameter and its asymptotic multivariate normal distribution. Our bounds apply in situations in which the MLE can be written as a function of a sum of i.i.d. $t$-dimensional random vectors. We apply our general bound to establish a bound for the multivariate normal approximation of the MLE of the normal distribution with unknown mean and variance.
We obtain explicit $p$-Wasserstein distance error bounds between the distribution of the multi-parameter MLE and the multivariate normal distribution. Our general bounds are given for possibly high-dimensional, independent and identically distributed random vectors. Our general bounds are of the optimal $\mathcal{O}(n^{-1/2})$ order. Explicit numerical constants are given when $p\in(1,2]$, and in the case $p>2$ the bounds are explicit up to a constant factor that only depends on $p$. We apply our general bounds to derive Wasserstein distance error bounds for the multivariate normal approximati…
We consider an inference on the eigenvalues of the covariance matrix of a multivariate normal distribution. The family of multivariate normal distributions with a fixed mean is seen as a Riemannian manifold with Fisher information metric. Two submanifolds naturally arises; one is the submanifold given by fixed eigenvectors of the covariance matrix, the other is the one given by fixed eigenvalues. We analyze the geometrical structures of these manifolds such as metric, embedding curvature under $e$-connection or $m$-connection. Based on these results, we study 1) the bias of the sample eigenval…
In this article we provide some nonnegative and positive estimators of the mean squared errors(MSEs) for shrinkage estimators of multivariate normal means. Proposed estimators are shown to improve on the uniformly minimum variance unbiased estimator(UMVUE) under a quadratic loss criterion. A similar improvement is also obtained for the estimators of the MSE matrices for shrinkage estimators. We also apply the proposed estimators of the MSE matrix to form confidence sets centered at shrinkage estimators and show their usefulness through numerical experiments.
In this work, the estimation of the multivariate normal mean by different classes of shrinkage estimators is investigated. The risk associated with the balanced loss function is used to compare two estimators. We start by considering estimators that generalize the James-Stein estimator and show that these estimators dominate the maximum likelihood estimator (MLE), therefore are minimax, when the shrinkage function satisfies some conditions. Then, we treat estimators of polynomial form and prove the increase of the degree of the polynomial allows us to build a better estimator from the one prev…
We consider the problem of estimating covariance and precision matrices, and their associated discriminant coefficients, from normal data when the rank of the covariance matrix is strictly smaller than its dimension and the available sample size. Using unbiased risk estimation, we construct novel estimators by minimizing upper bounds on the difference in risk over several classes. Our proposal estimates are empirically demonstrated to offer substantial improvement over classical approaches.
This paper mainly focuses on studying the Shannon Entropy and Kullback-Leibler divergence of the multivariate log canonical fundamental skew-normal (LCFUSN) and canonical fundamental skew-normal (CFUSN) families of distributions, extending previous works. We relate our results with other well known distributions entropies. As a byproduct, we also obtain the Mutual Information for distributions in these families. Shannon entropy is used to compare models fitted to analyze the USA monthly precipitation data. Kullback-Leibler divergence is used to cluster regions in Atlantic ocean according to th…
In this short note, we develop a local approximation for the log-ratio of the multivariate hypergeometric probability mass function over the corresponding multinomial probability mass function. In conjunction with the bounds from Carter (2002) and Ouimet (2021) on the total variation between the law of a multinomial vector jittered by a uniform on $(-1/2,1/2)^d$ and the law of the corresponding multivariate normal distribution, the local expansion for the log-ratio is then used to obtain a total variation bound between the law of a multivariate hypergeometric random vector jittered by a unifor…
Robust estimation under multivariate normal (MVN) mixture model is always a computational challenge. A recently proposed maximum pseudo \b{eta}-likelihood estimator aims to estimate the unknown parameters of a MVN mixture model in the spirit of minimum density power divergence (DPD) methodology but with a relatively simpler and tractable computational algorithm even for larger dimensions. In this letter, we will rigorously derive the existence and weak consistency of the maximum pseudo \b{eta}-likelihood estimator in case of MVN mixture models under a reasonable set of assumptions.
In this article, we consider two forms of shrinkage estimators of the mean $θ$ of a multivariate normal distribution $X\sim N_{p}\left(θ, σ^{2}I_{p}\right)$ where $σ^{2}$ is unknown. We take the prior law $θ\sim N_{p}\left(\upsilon, τ^{2}I_{p}\right)$ and we constuct a Modified Bayes estimator $δ_{B}^{\ast}$ and an Empirical Modified Bayes estimator $δ_{EB}^{\ast}$. We are interested in studying the minimaxity and the limits of risks ratios of these estimators, to the maximum likelihood estimator $X$, when $n$ and $p$ tend to infinity.
We show that the orthogonal projection operator onto the range of the adjoint of a linear operator $T$ can be represented as $UT,$ where $U$ is an invertible linear operator. Using this representation we obtain a decomposition of a Normal random vector $Y$ as the sum of a linear transformation of $Y$ that is independent of $TY$ and an affine transformation of $TY$. We then use this decomposition to prove that the conditional distribution of a Normal random vector $Y$ given a linear transformation $\mathcal{T}Y$ is again a multivariate Normal distribution. This result is equivalent to the well-…
The problem of characterizing a multivariate distribution of a random vector using examination of univariate combinations of vector components is an essential issue of multivariate analysis. The likelihood principle plays a prominent role in developing powerful statistical inference tools. In this context, we raise the question: can the univariate likelihood function based on a random vector be used to provide the uniqueness in reconstructing the vector distribution? In multivariate normal (MN) frameworks, this question links to a reverse of Cochran's theorem that concerns the distribution of …
The notion of entropy-regularized optimal transport, also known as Sinkhorn divergence, has recently gained popularity in machine learning and statistics, as it makes feasible the use of smoothed optimal transportation distances for data analysis. The Sinkhorn divergence allows the fast computation of an entropically regularized Wasserstein distance between two probability distributions supported on a finite metric space of (possibly) high-dimension. For data sampled from one or two unknown probability distributions, we derive the distributional limits of the empirical Sinkhorn divergence and …
In this paper, a novel Bayesian nonparametric test for assessing multivariate normal models is presented. While there are extensive frequentist and graphical methods for testing multivariate normality, it is challenging to find Bayesian counterparts. The proposed approach is based on the use of the Dirichlet process and Mahalanobis distance. More precisely, the Mahalanobis distance is employed as a good technique to transform the $m$-variate problem into a univariate problem. Then the Dirichlet process is used as a prior on the distribution of the Mahalanobis distance. The concentration of the…
We show that lower-dimensional marginal densities of dependent zero-mean normal distributions truncated to the positive orthant exhibit a mass-shifting phenomenon. Despite the truncated multivariate normal density having a mode at the origin, the marginal density assigns increasingly small mass near the origin as the dimension increases. The phenomenon accentuates with stronger correlation between the random variables. A precise quantification characterizing the role of the dimension as well as the dependence is provided. This surprising behavior has serious implications towards Bayesian const…
Multivariate normal mixtures provide a flexible method of fitting high-dimensional data. It is shown that their topography, in the sense of their key features as a density, can be analyzed rigorously in lower dimensions by use of a ridgeline manifold that contains all critical points, as well as the ridges of the density. A plot of the elevations on the ridgeline shows the key features of the mixed density. In addition, by use of the ridgeline, we uncover a function that determines the number of modes of the mixed density when there are two components being mixed. A followup analysis then give…
It is shown that a simple Dirichlet process mixture of multivariate normals offers Bayesian density estimation with adaptive posterior convergence rates. Toward this, a novel sieve for non-parametric mixture densities is explored, and its rate adaptability to various smoothness classes of densities in arbitrary dimension is demonstrated. This sieve construction is expected to offer a substantial technical advancement in studying Bayesian non-parametric mixture models based on stick-breaking priors.
We use the holonomic gradient method to evaluate the probability content of a simplex region under a multivariate normal distribution. This probability equals to the integral of the probability density function of the multivariate Gaussian distribution on the simplex region. For this purpose, we generalize the inclusion--exclusion identity which was given for polyhedra, to the faces of a polyhedron. This extended inclusion--exclusion identity enables us to calculate the derivatives of the function associated with the probability content of a polyhedron in general position. We show that these d…
The normalized maximum likelihood (NML) is one of the most important distribution in coding theory and statistics. NML is the unique solution (if exists) to the pointwise minimax regret problem. However, NML is not defined even for simple family of distributions such as the normal distributions. Since there does not exist any meaningful minimax-regret distribution for such case, it is pointed out that NML with luckiness (LNML) can be employed as an alternative to NML. In this paper, we develop the closed form of LNMLs for multivariate normal distributions.
We study a novel class of affine invariant and consistent tests for multivariate normality. The tests are based on a characterization of the standard $d$-variate normal distribution by means of the unique solution of an initial value problem connected to a partial differential equation, which is motivated by a multivariate Stein equation. The test criterion is a suitably weighted $L^2$-statistic. We derive the limit distribution of the test statistic under the null hypothesis as well as under contiguous and fixed alternatives to normality. A consistent estimator of the limiting variance under …
Finite mixtures of multivariate normal distributions have been widely used in empirical applications in diverse fields such as statistical genetics and statistical finance. Testing the number of components in multivariate normal mixture models is a long-standing challenge even in the most important case of testing homogeneity. This paper develops likelihood-based tests of the null hypothesis of $M_0$ components against the alternative hypothesis of $M_0 + 1$ components for a general $M_0 \geq 1$. For heteroscedastic normal mixtures, we propose an EM test and derive the asymptotic distribution …
In this paper, a new mixture family of multivariate normal distributions, formed by mixing multivariate normal distribution and skewed distribution, is constructed. Some properties of this family, such as characteristic function, moment generating function, and the first four moments are derived. The distributions of affine transformations and canonical forms of the model are also derived. An EM type algorithm is developed for the maximum likelihood estimation of model parameters. We have considered in detail, some special cases of the family, using standard gamma and standard exponential mixt…
In this short note, we prove an asymptotic expansion for the ratio of the Dirichlet density to the multivariate normal density with the same mean and covariance matrix. The expansion is then used to derive an upper bound on the total variation between the corresponding probability measures and rederive the asymptotic variance of the Dirichlet kernel estimators introduced by Aitchison & Lauder (1985) and studied theoretically in Ouimet (2020). Another potential application related to the asymptotic equivalence between the Gaussian variance regression problem and the Gaussian white noise problem…
We investigate predictive densities for multivariate normal models with unknown mean vectors and known covariance matrices. Bayesian predictive densities based on shrinkage priors often have complex representations, although they are effective in various problems. We consider extended normal models with mean vectors and covariance matrices as parameters, and adopt predictive densities that belong to the extended models including the original normal model. We adopt predictive densities that are optimal with respect to the posterior Bayes risk in the extended models. The proposed predictive dens…
Several formulations have long existed in the literature in the form of continuous mixtures of normal variables where a mixing variable operates on the mean or on the variance or on both the mean and the variance of a multivariate normal variable, by changing the nature of these basic constituents from constants to random quantities. More recently, other mixture-type constructions have been introduced, where the core random component, on which the mixing operation operates, is not necessarily normal. The main aim of the present work is to show that many existing constructions can be encompasse…
Multivariate normal mixtures provide a flexible model for high-dimensional data. They are widely used in statistical genetics, statistical finance, and other disciplines. Due to the unboundedness of the likelihood function, classical likelihood-based methods, which may have nice practical properties, are inconsistent. In this paper, we recommend a penalized likelihood method for estimating the mixing distribution. We show that the maximum penalized likelihood estimator is strongly consistent when the number of components has a known upper bound. We also explore a convenient EM-algorithm for co…
We study a novel class of affine invariant and consistent tests for normality in any dimension. The tests are based on a characterization of the standard $d$-variate normal distribution as the unique solution of an initial value problem of a partial differential equation motivated by the harmonic oscillator, which is a special case of a Schrödinger operator. We derive the asymptotic distribution of the test statistics under the hypothesis of normality as well as under fixed and contiguous alternatives. The tests are consistent against general alternatives, exhibit strong power performance for …
In this paper, the multivariate tail covariance (MTCov) for generalized skew-elliptical distributions is considered. Some special cases for this distribution, such as generalized skew-normal, generalized skew student-t, generalized skew-logistic and generalized skew-Laplace distributions, are also considered. In order to test the theoretical feasibility of our results, the MTCov for skewed and non skewed normal distributions are computed and compared. Finally, we give a special formula of the MTCov for generalized skew-elliptical distributions.
Let $μ$ be a Gaussian measure (say, on ${\bf R}^n$) and let $K, L \subset {\bf R}^n$ be such that K is convex, $L$ is a "layer" (i.e. $L = \{x : a \leq < x,u > \leq b \}$ for some $a$, $b \in {\bf R}$ and $u \in {\bf R}^n$) and the centers of mass (with respect to $μ$) of $K$ and $L$ coincide. Then $μ(K \cap L) \geq μ(K) \cdot μ(L)$. This is motivated by the well-known "positive correlation conjecture" for symmetric sets and a related inequality of Sidak concerning confidence regions for means of multivariate normal distributions. The proof uses an apparently hitherto unknown estimate for the …
