Estimating the unconstrained mean and covariance matrix is a popular topic in statistics. However, estimation of the parameters of $N_p(μ,Σ)$ under joint constraints such as $Σμ= μ$ has not received much attention. It can be viewed as a multivariate counterpart of the classical estimation problem in the $N(θ,θ^2)$ distribution. In addition to the usual inference challenges under such non-linear constraints among the parameters (curved exponential family), one has to deal with the basic requirements of symmetry and positive definiteness when estimating a covariance matrix. We derive the non-linear likelihood equations for the constrained maximum likelihood estimator of $(μ,Σ)$ and solve them using iterative methods. Generally, the MLE of covariance matrices computed using iterative methods do not satisfy the constraints. We propose a novel algorithm to modify such (infeasible) estimators or any other (reasonable) estimator. The key step is to re-align the mean vector along the eigenvectors of the covariance matrix using the idea of regression. In using the Lagrangian function for constrained MLE (Aitchison et al. 1958), the Lagrange multiplier entangles with the parameters of interest and presents another computational challenge. We handle this by either iterative or explicit calculation of the Lagrange multiplier. The existence and nature of location of the constrained MLE are explored within a data-dependent convex set using recent results from random matrix theory. A simulation study illustrates our methodology and shows that the modified estimators perform better than the initial estimators from the iterative methods.
We establish global rates of convergence of the Maximum Likelihood Estimator (MLE) of a multivariate distribution function in the case of (one type of) "interval censored" data. The main finding is that the rate of convergence of the MLE in the Hellinger metric is no worse than $n^{-1/3} (\log n)^γ$ for $γ= (5d - 4)/6$.
We study the distribution of the maximum likelihood estimate (MLE) in high-dimensional logistic models, extending the recent results from Sur (2019) to the case where the Gaussian covariates may have an arbitrary covariance structure. We prove that in the limit of large problems holding the ratio between the number $p$ of covariates and the sample size $n$ constant, every finite list of MLE coordinates follows a multivariate normal distribution. Concretely, the $j$th coordinate $\hat β_j$ of the MLE is asymptotically normally distributed with mean $α_\star β_j$ and standard deviation $σ_\star/…
We study a non-parametric approach to multivariate density estimation. The estimators are piecewise constant density functions supported by binary partitions. The partition of the sample space is learned by maximizing the likelihood of the corresponding histogram on that partition. We analyze the convergence rate of the sieve maximum likelihood estimator, and reach a conclusion that for a relatively rich class of density functions the rate does not directly depend on the dimension. This suggests that, under certain conditions, this method is immune to the curse of dimensionality, in the sense …
Sample covariance matrices are widely used in multivariate statistical analysis. The central limit theorems (CLT's) for linear spectral statistics of high-dimensional non-centered sample covariance matrices have received considerable attention in random matrix theory and have been applied to many high-dimensional statistical problems. However, known population mean vectors are assumed for non-centered sample covariance matrices, some of which even assume Gaussian-like moment conditions. In fact, there are still another two most frequently used sample covariance matrices: the MLE (by subtractin…
The asymptotic normality of the maximum likelihood estimator (MLE) under regularity conditions is a cornerstone of statistical theory. In this paper, we give explicit upper bounds on the distributional distance between the distribution of the MLE of a vector parameter, and the multivariate normal distribution. We work with possibly high-dimensional, independent but not necessarily identically distributed random vectors. In addition, we obtain explicit upper bounds even in cases where the MLE cannot be expressed analytically.
In matrix-valued datasets the sampled matrices often exhibit correlations among both their rows and their columns. A useful and parsimonious model of such dependence is the matrix normal model, in which the covariances among the elements of a random matrix are parameterized in terms of the Kronecker product of two covariance matrices, one representing row covariances and one representing column covariance. An appealing feature of such a matrix normal model is that the Kronecker covariance structure allows for standard likelihood inference even when only a very small number of data matrices is …
We use the delta method and Stein's method to derive, under regularity conditions, explicit upper bounds for the distributional distance between the distribution of the maximum likelihood estimator (MLE) of a $d$-dimensional parameter and its asymptotic multivariate normal distribution. Our bounds apply in situations in which the MLE can be written as a function of a sum of i.i.d. $t$-dimensional random vectors. We apply our general bound to establish a bound for the multivariate normal approximation of the MLE of the normal distribution with unknown mean and variance.
Structured Latent Attribute Models (SLAMs) are a family of discrete latent variable models widely used in education, psychology, and epidemiology to model multivariate categorical data. A SLAM assumes that multiple discrete latent attributes explain the dependence of observed variables in a highly structured fashion. Usually, the maximum marginal likelihood estimation approach is adopted for SLAMs, treating the latent attributes as random effects. The increasing scope of modern assessment data involves large numbers of observed variables and high-dimensional latent attributes. This poses chall…
We obtain explicit $p$-Wasserstein distance error bounds between the distribution of the multi-parameter MLE and the multivariate normal distribution. Our general bounds are given for possibly high-dimensional, independent and identically distributed random vectors. Our general bounds are of the optimal $\mathcal{O}(n^{-1/2})$ order. Explicit numerical constants are given when $p\in(1,2]$, and in the case $p>2$ the bounds are explicit up to a constant factor that only depends on $p$. We apply our general bounds to derive Wasserstein distance error bounds for the multivariate normal approximati…
We develop a new method to fit the multivariate response linear regression model that exploits a parametric link between the regression coefficient matrix and the error covariance matrix. Specifically, we assume that the correlations between entries in the multivariate error random vector are proportional to the cosines of the angles between their corresponding regression coefficient matrix columns, so as the angle between two regression coefficient matrix columns decreases, the correlation between the corresponding errors increases. We highlight two models under which this parameterization ar…
We consider covariance estimation in the multivariate generalized Gaussian distribution (MGGD) and elliptically symmetric (ES) distribution. The maximum likelihood optimization associated with this problem is non-convex, yet it has been proved that its global solution can be often computed via simple fixed point iterations. Our first contribution is a new analysis of this likelihood based on geodesic convexity that requires weaker assumptions. Our second contribution is a generalized framework for structured covariance estimation under sparsity constraints. We show that the optimizations can b…
When the spatial sample size is extremely large, which occurs in many environmental and ecological studies, operations on the large covariance matrix are a numerical challenge. Covariance tapering is a technique to alleviate the numerical challenges. Under the assumption that data are collected along a line in a bounded region, we investigate how the tapering affects the asymptotic efficiency of the maximum likelihood estimator (MLE) for the microergodic parameter in the Matérn covariance function by establishing the fixed-domain asymptotic distribution of the exact MLE and that of the tapered…
The prevalence of spatially referenced multivariate data has impelled researchers to develop a procedure for the joint modeling of multiple spatial processes. This ordinarily involves modeling marginal and cross-process dependence for any arbitrary pair of locations using a multivariate spatial covariance function. However, building a flexible multivariate spatial covariance function that is nonnegative definite is challenging. Here, we propose a semiparametric approach for multivariate spatial covariance function estimation with approximate Matérn marginals and highly flexible cross-covarianc…
Inference about dependencies in a multiway data array can be made using the array normal model, which corresponds to the class of multivariate normal distributions with separable covariance matrices. Maximum likelihood and Bayesian methods for inference in the array normal model have appeared in the literature, but there have not been any results concerning the optimality properties of such estimators. In this article, we obtain results for the array normal model that are analogous to some classical results concerning covariance estimation for the multivariate normal model. We show that under …
In this paper, we study a class of non-parametric density estimators under Bayesian settings. The estimators are piecewise constant functions on binary partitions. We analyze the concentration rate of the posterior distribution under a suitable prior, and demonstrate that the rate does not directly depend on the dimension of the problem. This paper can be viewed as an extension of a parallel work where the convergence rate of a related sieve MLE was established. Compared to the sieve MLE, the main advantage of the Bayesian method is that it can adapt to the unknown complexity of the true densi…
The Ohio State University（俄亥俄州立大学）；University of Southern California（南加州大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
We study the problem of learning multivariate log-concave densities with respect to a global loss function. We obtain the first upper bound on the sample complexity of the maximum likelihood estimator (MLE) for a log-concave density on $\mathbb{R}^d$, for all $d \geq 4$. Prior to this work, no finite sample upper bound was known for this estimator in more than $3$ dimensions. In more detail, we prove that for any $d \geq 1$ and $ε>0$, given $\tilde{O}_d((1/ε)^{(d+3)/2})$ samples drawn from an unknown log-concave density $f_0$ on $\mathbb{R}^d$, the MLE outputs a hypothesis $h$ that with high p…
In this paper, we introduce a new multivariate discrete distribution which called multivariate unification of generalized Apostol type distribution (UGAT). Several prop- erties are studied as, moments, probability generating function and other properties. Also, reliability study of distribution are introduced. Maximum likelihood method is used to estimate parameters and numerical method is used to obtain (MLEs).
Suppose that $\m{U} = (U_1, \ldots , U_d) $ has a Uniform$([0,1]^d)$ distribution, that $\m{Y} = (Y_1 , \ldots , Y_d) $ has the distribution $G$ on $\RR_+^d$, and let $\m{X} = (X_1 , \ldots , X_d) = (U_1 Y_1 , \ldots , U_d Y_d )$. The resulting class of distributions of $\m{X}$ (as $G$ varies over all distributions on $\RR_+^d$) is called the {\sl Scale Mixture of Uniforms} class of distributions, and the corresponding class of densities on $\RR_+^d$ is denoted by $\{\cal F}_{SMU}(d)$. We study maximum likelihood estimation in the family ${\cal F}_{SMU}(d)$. We prove existence of the MLE, esta…
Due to its heavy-tailed and fully parametric form, the multivariate generalized Gaussian distribution (MGGD) has been receiving much attention for modeling extreme events in signal and image processing applications. Considering the estimation issue of the MGGD parameters, the main contribution of this paper is to prove that the maximum likelihood estimator (MLE) of the scatter matrix exists and is unique up to a scalar factor, for a given shape parameter β\in(0,1). Moreover, an estimation algorithm based on a Newton-Raphson recursion is proposed for computing the MLE of MGGD parameters. Variou…
The Behrens-Fisher problem is a well-known hypothesis testing problem in statistics concerning two-sample mean comparison. In this article, we confirm one conjecture in Eaton and Olshen (1972), which provides stochastic bounds for the multivariate Behrens-Fisher test statistic under the null hypothesis. We also extend their results on the stochastic ordering of random quotients to the arbitrary finite dimensional case. This work can also be seen as a generalization of Hsu (1938) that provided the bounds for the univariate Behrens-Fisher problem. The results obtained in this article can be used…
The L1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix, or alternatively the underlying graph structure of a Gaussian Markov Random Field, from very limited samples. We propose a novel algorithm for solving the resulting optimization problem which is a regularized log-determinant program. In contrast to recent state-of-the-art methods that largely use first order gradient information, our algorithm is based on Newton's method and employs a quadratic approximation, but with some modific…
The weak variance-alpha-gamma process is a multivariate Lévy process constructed by weakly subordinating Brownian motion, possibly with correlated components with an alpha-gamma subordinator. It generalises the variance-alpha-gamma process of Semeraro constructed by traditional subordination. We compare three calibration methods for the weak variance-alpha-gamma process, method of moments, maximum likelihood estimation (MLE) and digital moment estimation (DME). We derive a condition for Fourier invertibility needed to apply MLE and show in our simulations that MLE produces a better fit when th…
Multivariate processes with long-range dependence properties can be encountered in many fields of application. Two fundamental characteristics in such frameworks are long-range dependence parameters and correlations between component time series. We consider multivariate long-range dependent linear processes, not necessarily Gaussian. We show that the covariances between the wavelet coefficients in this setting are asymptotically Gaussian. We also study the asymptotic distributions of the estimators of the long-range dependence parameter and the long-run covariance by a wavelet-based Whittle p…
We consider goodness-of-fit tests of symmetric stable distributions based on weighted integrals of the squared distance between the empirical characteristic function of the standardized data and the characteristic function of the standard symmetric stable distribution with the characteristic exponent $α$ estimated from the data. We treat $α$ as an unknown parameter, but for theoretical simplicity we also consider the case that $α$ is fixed. For estimation of parameters and the standardization of data we use maximum likelihood estimator (MLE) and an equivariant integrated squared error estimato…
We propose three measures of mutual dependence between multiple random vectors. All the measures are zero if and only if the random vectors are mutually independent. The first measure generalizes distance covariance from pairwise dependence to mutual dependence, while the other two measures are sums of squared distance covariance. All the measures share similar properties and asymptotic distributions to distance covariance, and capture non-linear and non-monotone mutual dependence between the random vectors. Inspired by complete and incomplete V-statistics, we define the empirical measures and…
This paper considers the problem of robustly estimating the parameters of a heavy-tailed multivariate distribution when the covariance matrix is known to have the structure of a low-rank matrix plus a diagonal matrix as considered in factor analysis (FA). By assuming the observed data to follow the multivariate Student's t distribution, we can robustly estimate the parameters via maximum likelihood estimation (MLE). However, the MLE of parameters becomes an intractable problem when the multivariate Student's t distribution and the FA structure are both introduced. In this paper, we propose an …
There has been a growing interest in providing models for multivariate spatial processes. A majority of these models specify a parametric matrix covariance function. Based on observations, the parameters are estimated by maximum likelihood or variants thereof. While the asymptotic properties of maximum likelihood estimators for univariate spatial processes have been analyzed in detail, maximum likelihood estimators for multivariate spatial processes have not received their deserved attention yet. In this article we consider the classical increasing-domain asymptotic setting restricting the min…
This article presents maximum likelihood estimators (MLEs) and log-likelihood ratio (LLR) tests for the eigenvalues and eigenvectors of Gaussian random symmetric matrices of arbitrary dimension, where the observations are independent repeated samples from one or two populations. These inference problems are relevant in the analysis of diffusion tensor imaging data and polarized cosmic background radiation data, where the observations are, respectively, $3\times3$ and $2\times2$ symmetric positive definite matrices. The parameter sets involved in the inference problems for eigenvalues and eigen…
In this work, the estimation of the multivariate normal mean by different classes of shrinkage estimators is investigated. The risk associated with the balanced loss function is used to compare two estimators. We start by considering estimators that generalize the James-Stein estimator and show that these estimators dominate the maximum likelihood estimator (MLE), therefore are minimax, when the shrinkage function satisfies some conditions. Then, we treat estimators of polynomial form and prove the increase of the degree of the polynomial allows us to build a better estimator from the one prev…
We consider estimation of the covariance matrix of a multivariate random vector under the constraint that certain covariances are zero. We first present an algorithm, which we call Iterative Conditional Fitting, for computing the maximum likelihood estimator of the constrained covariance matrix, under the assumption of multivariate normality. In contrast to previous approaches, this algorithm has guaranteed convergence properties. Dropping the assumption of multivariate normality, we show how to estimate the covariance matrix in an empirical likelihood approach. These approaches are then compa…
We study exponential families of distributions that are multivariate totally positive of order 2 (MTP2), show that these are convex exponential families, and derive conditions for existence of the MLE. Quadratic exponential familes of MTP2 distributions contain attractive Gaussian graphical models and ferromagnetic Ising models as special examples. We show that these are defined by intersecting the space of canonical parameters with a polyhedral cone whose faces correspond to conditional independence relations. Hence MTP2 serves as an implicit regularizer for quadratic exponential families and…
We derive exact computable expressions for the asymptotic distribution of the change-point mle when a change in the mean occurred at an unknown point of a sequence of time-ordered independent Gaussian random variables. The derivation, which assumes that nuisance parameters such as the amount of change and variance are known, is based on ladder heights of Gaussian random walks hitting the half-line. We then show that the exact distribution easily extends to the distribution of the change-point mle when a change occurs in the mean vector of a multivariate Gaussian process. We perform simulations…
We provide a unified approach to a method of estimation of the regression parameter in balanced linear models with a structured covariance matrix that combines a high breakdown point and bounded influence with high asymptotic efficiency at models with multivariate normal errors. Of main interest are linear mixed effects models, but our approach also includes several other standard multivariate models, such as multiple regression, multivariate regression, and multivariate location and scatter. We provide sufficient conditions for the existence of the estimators and corresponding functionals, es…
In group sequential analysis, data is collected and analyzed in batches until pre-defined stopping criteria are met. Inference in the parametric setup typically relies on the limiting asymptotic multivariate normality of the repeatedly computed maximum likelihood estimators (MLEs), a result first rigorously proved by Jennison and Turbull (1997) under general regularity conditions. In this work, using Stein's method we provide optimal order, non-asymptotic bounds on the distance for smooth test functions between the joint group sequential MLEs and the appropriate normal distribution under the s…
In this paper, we present a novel framework incorporating a combination of sparse models in different domains. We posit the observed data as generated from a linear combination of a sparse Gaussian Markov model (with a sparse precision matrix) and a sparse Gaussian independence model (with a sparse covariance matrix). We provide efficient methods for decomposition of the data into two domains, \viz Markov and independence domains. We characterize a set of sufficient conditions for identifiability and model consistency. Our decomposition method is based on a simple modification of the popular $…
In arXiv:0907.0079 by Cator and Lopuhaa, an asymptotic expansion for the MCD estimators is established in a very general framework. This expansion requires the existence and non-singularity of the derivative in a first-order Taylor expansion. In this paper, we prove the existence of this derivative for multivariate distributions that have a density and provide an explicit expression. Moreover, under suitable symmetry conditions on the density, we show that this derivative is non-singular. These symmetry conditions include the elliptically contoured multivariate location-scatter model, in which…
We provide a unified approach to S-estimation in balanced linear models with structured covariance matrices. Of main interest are S-estimators for linear mixed effects models, but our approach also includes S-estimators in several other standard multivariate models, such as multiple regression, multivariate regression, and multivariate location and scatter. We provide sufficient conditions for the existence of S-functionals and S-estimators, establish asymptotic properties such as consistency and asymptotic normality, and derive their robustness properties in terms of breakdown point and influ…
Asymptotic distribution for the proportional covariance model under multivariate normal distributions is derived. To this end, the parametrization of the common covariance matrix by its Cholesky root is adopted. The derivations are made in three steps. First, the asymptotic distribution of the maximum likelihood estimators of the proportionality coefficients and the Cholesky inverse root of the common covariance matrix is derived by finding the information matrix and its inverse. Next, the asymptotic distributions for the case of the Cholesky root of the common covariance matrix and finally fo…
Markov regime switching models have been widely used in numerous empirical applications in economics and finance. However, the asymptotic distribution of the maximum likelihood estimator (MLE) has not been proven for some empirically popular Markov regime switching models. In particular, the asymptotic distribution of the MLE has been unknown for models in which some elements of the transition probability matrix have the value of zero, as is commonly assumed in empirical applications with models with more than two regimes. This also includes models in which the regime-specific density depends …
We define the minimum covariance determinant functionals for multivariate location and scatter through trimming functions and establish their existence at any multivariate distribution. We provide a precise characterization including a separating ellipsoid property and prove that the functionals are continuous. Moreover, we establish asymptotic normality for both the location and covariance estimator and derive the influence function. These results are obtained in a very general multivariate setting.
It is well known that under general regularity conditions the distribution of the maximum likelihood estimator (MLE) is asymptotically normal. Very recently, bounds of the optimal order $O(1/\sqrt n)$ on the closeness of the distribution of the MLE to normality in the so-called bounded Wasserstein distance were obtained, where $n$ is the sample size. However, the corresponding bounds on the Kolmogorov distance were only of the order $O(1/n^{1/4})$. In this note, bounds of the optimal order $O(1/\sqrt n)$ on the closeness of the distribution of the MLE to normality in the Kolmogorov distance ar…
For normal canonical models with $X \sim N_p(θ, σ^{2} I_{p}), \;\; S^{2} \sim σ^{2}χ^{2}_{k}, \;{independent}$, we consider the problem of estimating $θ$ under scale invariant squared error loss $\frac{\|d-θ\|^{2}}{σ^{2}}$, when it is known that the signal-to-noise ratio $\frac{\|θ\|}σ$ is bounded above by $m$. Risk analysis is achieved by making use of a conditional risk decomposition and we obtain in particular sufficient conditions for an estimator to dominate either the unbiased estimator $δ_{UB}(X)=X$, or the maximum likelihood estimator $δ_{\hbox{mle}}(X,S^2)$, or both of these benchmark…
The integration of physical relationships into stochastic models is of major interest e.g. in data assimilation. Here, a multivariate Gaussian random field formulation is introduced, which represents the differential relations of the two-dimensional wind field and related variables such as streamfunction, velocity potential, vorticity and divergence. The covariance model is based on a flexible bivariate Matérn covariance function for streamfunction and velocity potential. It allows for different variances in the potentials, non-zero correlations between them, anisotropy and a flexible smoothne…
