In matrix-valued datasets the sampled matrices often exhibit correlations among both their rows and their columns. A useful and parsimonious model of such dependence is the matrix normal model, in which the covariances among the elements of a random matrix are parameterized in terms of the Kronecker product of two covariance matrices, one representing row covariances and one representing column covariance. An appealing feature of such a matrix normal model is that the Kronecker covariance structure allows for standard likelihood inference even when only a very small number of data matrices is available. For instance, in some cases a likelihood ratio test of dependence may be performed with a sample size of one. However, more generally the sample size required to ensure boundedness of the matrix normal likelihood or the existence of a unique maximizer depends in a complicated way on the matrix dimensions. This motivates the study of how large a sample size is needed to ensure that maximum likelihood estimators exist, and exist uniquely with probability one. Our main result gives precise sample size thresholds in the paradigm where the number of rows and the number of columns of the data matrices differ by at most a factor of two. Our proof uses invariance properties that allow us to consider data matrices in canonical form, as obtained from the Kronecker canonical form for matrix pencils.
We study the distribution of the maximum likelihood estimate (MLE) in high-dimensional logistic models, extending the recent results from Sur (2019) to the case where the Gaussian covariates may have an arbitrary covariance structure. We prove that in the limit of large problems holding the ratio between the number $p$ of covariates and the sample size $n$ constant, every finite list of MLE coordinates follows a multivariate normal distribution. Concretely, the $j$th coordinate $\hat β_j$ of the MLE is asymptotically normally distributed with mean $α_\star β_j$ and standard deviation $σ_\star/…
This paper deals with multivariate Gaussian models for which the covariance matrix is a Kronecker product of two matrices. We consider maximum likelihood estimation of the model parameters, in particular of the covariance matrix. There is no explicit expression for the maximum likelihood estimator of a Kronecker product covariance matrix. The main question in this paper is whether the maximum likelihood estimator of the covariance matrix exists and if it is unique. The answers are different for different models that we consider.
Kronecker product covariance structure provides an efficient way to modeling the inter-correlations of matrix-variate data. In this paper, we propose testing statistics for Kronecker product covariance matrix based on linear spectral statistics of renormalized sample covariance matrices. Central limit theorem is proved for the linear spectral statistics with explicit formulas for mean and covariance functions, which fills the gap in the literature. We then theoretically justify that the proposed testing statistics have well-controlled sizes and strong powers. To facilitate practical usefulness…
The matrix-variate normal distribution is a popular model for high-dimensional transposable data because it decomposes the dependence structure of the random matrix into the Kronecker product of two covariance matrices: one for each of the row and column variables. We develop tests for assessing the form of the row (column) covariance matrix in high-dimensional settings while treating the column (row) dependence structure as a nuisance. Our tests are robust to normality departures provided that the Kronecker product dependence structure holds. In simulations, we observe that the proposed tests…
Estimating the unconstrained mean and covariance matrix is a popular topic in statistics. However, estimation of the parameters of $N_p(μ,Σ)$ under joint constraints such as $Σμ= μ$ has not received much attention. It can be viewed as a multivariate counterpart of the classical estimation problem in the $N(θ,θ^2)$ distribution. In addition to the usual inference challenges under such non-linear constraints among the parameters (curved exponential family), one has to deal with the basic requirements of symmetry and positive definiteness when estimating a covariance matrix. We derive the non-lin…
When the spatial sample size is extremely large, which occurs in many environmental and ecological studies, operations on the large covariance matrix are a numerical challenge. Covariance tapering is a technique to alleviate the numerical challenges. Under the assumption that data are collected along a line in a bounded region, we investigate how the tapering affects the asymptotic efficiency of the maximum likelihood estimator (MLE) for the microergodic parameter in the Matérn covariance function by establishing the fixed-domain asymptotic distribution of the exact MLE and that of the tapered…
Inference about dependencies in a multiway data array can be made using the array normal model, which corresponds to the class of multivariate normal distributions with separable covariance matrices. Maximum likelihood and Bayesian methods for inference in the array normal model have appeared in the literature, but there have not been any results concerning the optimality properties of such estimators. In this article, we obtain results for the array normal model that are analogous to some classical results concerning covariance estimation for the multivariate normal model. We show that under …
We propose a Kronecker product model for correlation or covariance matrices in the large dimensional case. The number of parameters of the model increases logarithmically with the dimension of the matrix. We propose a minimum distance (MD) estimator based on a log-linear property of the model, as well as a one-step estimator, which is a one-step approximation to the quasi-maximum likelihood estimator (QMLE). We establish rates of convergence and central limit theorems (CLT) for our estimators in the large dimensional case. A specification test and tools for Kronecker product model selection an…
Sample covariance matrices are widely used in multivariate statistical analysis. The central limit theorems (CLT's) for linear spectral statistics of high-dimensional non-centered sample covariance matrices have received considerable attention in random matrix theory and have been applied to many high-dimensional statistical problems. However, known population mean vectors are assumed for non-centered sample covariance matrices, some of which even assume Gaussian-like moment conditions. In fact, there are still another two most frequently used sample covariance matrices: the MLE (by subtractin…
This article presents maximum likelihood estimators (MLEs) and log-likelihood ratio (LLR) tests for the eigenvalues and eigenvectors of Gaussian random symmetric matrices of arbitrary dimension, where the observations are independent repeated samples from one or two populations. These inference problems are relevant in the analysis of diffusion tensor imaging data and polarized cosmic background radiation data, where the observations are, respectively, $3\times3$ and $2\times2$ symmetric positive definite matrices. The parameter sets involved in the inference problems for eigenvalues and eigen…
Markov regime switching models have been widely used in numerous empirical applications in economics and finance. However, the asymptotic distribution of the maximum likelihood estimator (MLE) has not been proven for some empirically popular Markov regime switching models. In particular, the asymptotic distribution of the MLE has been unknown for models in which some elements of the transition probability matrix have the value of zero, as is commonly assumed in empirical applications with models with more than two regimes. This also includes models in which the regime-specific density depends …
The L1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix, or alternatively the underlying graph structure of a Gaussian Markov Random Field, from very limited samples. We propose a novel algorithm for solving the resulting optimization problem which is a regularized log-determinant program. In contrast to recent state-of-the-art methods that largely use first order gradient information, our algorithm is based on Newton's method and employs a quadratic approximation, but with some modific…
Near optimal sample complexity for matrix and tensor normal models via geodesic convexity
Department of Mathematics, Massachusetts Institute of Technology（麻省理工学院数学系）；Cheriton School of Computer Science, University of Waterloo（滑铁卢大学丘成桐计算机科学学院）；Computer Science Department, University of British Columbia（不列颠哥伦比亚大学计算机科学系）；Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学）
We analyze the problem of maximum likelihood estimation for Gaussian distributions that are multivariate totally positive of order two (MTP2). By exploiting connections to phylogenetics and single-linkage clustering, we give a simple proof that the maximum likelihood estimator (MLE) for such distributions exists based on at least 2 observations, irrespective of the underlying dimension. Slawski and Hein, who first proved this result, also provided empirical evidence showing that the MTP2 constraint serves as an implicit regularizer and leads to sparsity in the estimated inverse covariance matr…
Nonsingular estimation of high dimensional covariance matrices is an important step in many statistical procedures like classification, clustering, variable selection an future extraction. After a review of the essential background material, this paper introduces a technique we call slicing for obtaining a nonsingular covariance matrix of high dimensional data. Slicing is essentially assuming that the data has Kronecker delta covariance structure. Finally, we discuss the implications of the results in this paper and provide an example of classification for high dimensional gene expression data…
We propose a new estimator, the quadratic form estimator, of the Kronecker product model for covariance matrices. We show that this estimator has good properties in the large dimensional case (i.e., the cross-sectional dimension $n$ is large relative to the sample size $T$). In particular, the quadratic form estimator is consistent in a relative Frobenius norm sense provided $\log^3n/T\to 0$. We obtain the limiting distributions of Lagrange multiplier (LM) and Wald tests under both the null and local alternatives concerning the mean vector $μ$. Testing linear restrictions of $μ$ is also invest…
In this paper, we present a novel framework incorporating a combination of sparse models in different domains. We posit the observed data as generated from a linear combination of a sparse Gaussian Markov model (with a sparse precision matrix) and a sparse Gaussian independence model (with a sparse covariance matrix). We provide efficient methods for decomposition of the data into two domains, \viz Markov and independence domains. We characterize a set of sufficient conditions for identifiability and model consistency. Our decomposition method is based on a simple modification of the popular $…
Maximum likelihood estimation is a popular method in statistical inference. As a way of assessing the accuracy of the maximum likelihood estimate (MLE), the calculation of the covariance matrix of the MLE is of great interest in practice. Standard statistical theory shows that the normalized MLE is asymptotically normally distributed with covariance matrix being the inverse of the Fisher information matrix (FIM) at the unknown parameter. Two commonly used estimates for the covariance of the MLE are the inverse of the observed FIM (the same as the inverse Hessian of the negative log-likelihood)…
Covariance alignment: from maximum likelihood estimation to Gromov-Wasserstein
Center for Data Science, New York University（纽约大学数据科学中心）；Institute for Data, Systems, and Society, Massachusetts Institute of Technology（麻省理工学院数据、系统与社会研究所）；Department of Mathematics, Massachusetts Institute of Technology（麻省理工学院数学系）
Fitting high-dimensional data involves a delicate tradeoff between faithful representation and the use of sparse models. Too often, sparsity assumptions on the fitted model are too restrictive to provide a faithful representation of the observed data. In this paper, we present a novel framework incorporating sparsity in different domains.We decompose the observed covariance matrix into a sparse Gaussian Markov model (with a sparse precision matrix) and a sparse independence model (with a sparse covariance matrix). Our framework incorporates sparse covariance and sparse precision estimation as …
In this article, we introduce a two-way factor model for a high-dimensional data matrix and study the properties of the maximum likelihood estimation (MLE). The proposed model assumes separable effects of row and column attributes and captures the correlation across rows and columns with low-dimensional hidden factors. The model inherits the dimension-reduction feature of classical factor models but introduces a new framework with separable row and column factors, representing the covariance or correlation structure in the data matrix. We propose a block alternating, maximizing strategy to com…
Standard statistical methods applied to matrix random variables often fail to describe the underlying structure in multiway data sets. In this paper we will discuss the concept of an array variate random variable and introduce a class of elliptical array densities which have elliptical contours.
This paper develops a new class of conditional Markov jump processes with regime switching and paths dependence. The key novel feature of the developed process lies on its ability to switch the transition rate as it moves from one state to another with switching probability depending on the current state and time of the process as well as its past trajectories. As such, the transition from current state to another depends on the holding time of the process in the state. Distributional properties of the process are given explicitly in terms of the speed regimes represented by a finite number of…
In this paper, we will discuss the concept of an array variate random variable and introduce a class of skew normal array densities that are obtained through a selection model that uses the array variate normal density as the kernel and the cumulative distribution of the univariate normal distribution as the selection function.
In this paper, we mainly focus on the penalized maximum likelihood estimation (MLE) of the high-dimensional approximate factor model. Since the current estimation procedure can not guarantee the positive definiteness of the error covariance matrix, by reformulating the estimation of error covariance matrix and based on the lagrangian duality, we propose an accelerated proximal gradient (APG) algorithm to give a positive definite estimate of the error covariance matrix. Combined the APG algorithm with EM method, a new estimation procedure is proposed to estimate the high-dimensional approximate…
We explore the performance of sample average approximation in comparison with several other methods for stochastic optimization when there is information available on the underlying true probability distribution. The methods we evaluate are (a) bagging; (b) kernel smoothing; (c) maximum likelihood estimation (MLE); and (d) a Bayesian approach. We use two test sets, the first has a quadratic objective function allowing for very different types of interaction between the random component and the univariate decision variable. Here the sample average approximation is remarkably effective and only …
In this paper we introduce a covariance framework for the analysis of EEG and MEG data that takes into account observed temporal stationarity on small time scales and trial-to-trial variations. We formulate a model for the covariance matrix, which is a Kronecker product of three components that correspond to space, time and epochs/trials, and consider maximum likelihood estimation of the unknown parameter values. An iterative algorithm that finds approximations of the maximum likelihood estimates is proposed. We perform a simulation study to assess the performance of the estimator and investig…
This paper considers the problem of robustly estimating the parameters of a heavy-tailed multivariate distribution when the covariance matrix is known to have the structure of a low-rank matrix plus a diagonal matrix as considered in factor analysis (FA). By assuming the observed data to follow the multivariate Student's t distribution, we can robustly estimate the parameters via maximum likelihood estimation (MLE). However, the MLE of parameters becomes an intractable problem when the multivariate Student's t distribution and the FA structure are both introduced. In this paper, we propose an …
In the present paper, progression of non-alcoholic fatty liver disease (NAFLD) process is modeled by Continuous time Markov chains (CTMC) with 4 states .The transition intensities among the states are estimated using maximum likelihood estimation (MLE) method. The transition probabilities are also calculated. The mean sojourn time and its variance are estimated as well as the state probability distribution and its asymptotic covariance matrix. The life expectancy of the patient, one of the important statistical indices, is also obtained. The paper illustrates the new approach of using MLE to c…
Graphical models are a framework for representing and exploiting prior conditional independence structures within distributions using graphs. In the Gaussian case, these models are directly related to the sparsity of the inverse covariance (concentration) matrix and allow for improved covariance estimation with lower computational complexity. We consider concentration estimation with the mean-squared error (MSE) as the objective, in a special type of model known as decomposable. This model includes, for example, the well known banded structure and other cases encountered in practice. Our first…
In regression models for spatial data, it is often assumed that the marginal effects of covariates on the response are constant over space. In practice, this assumption might often be questionable. In this article, we show how a Gaussian process-based spatially varying coefficient (SVC) model can be estimated using maximum likelihood estimation (MLE). In addition, we present an approach that scales to large data by applying covariance tapering. We compare our methodology to existing methods such as a Bayesian approach using the stochastic partial differential equation (SPDE) link, geographical…
Hierarchical autocorrelation in the error term of linear models arises when sampling units are related to each other according to a tree. The residual covariance is parametrized using the tree-distance between sampling units. When observations are modeled using an Ornstein-Uhlenbeck (OU) process along the tree, the autocorrelation between two tips decreases exponentially with their tree distance. These models are most often applied in evolutionary biology, when tips represent biological species and the OU process parameters represent the strength and direction of natural selection. For these m…
The robust improper maximum likelihood estimator (RIMLE) is a new method for robust multivariate clustering finding approximately Gaussian clusters. It maximizes a pseudo-likelihood defined by adding a component with improper constant density for accommodating outliers to a Gaussian mixture. A special case of the RIMLE is MLE for multivariate finite Gaussian mixture models. In this paper we treat existence, consistency, and breakdown theory for the RIMLE comprehensively. RIMLE's existence is proved under non-smooth covariance matrix constraints. It is shown that these can be implemented via a …
Mohamed bin Zayed University of Artificial Intelligence（莫莫德·本·泽德人工智能大学）；National Center of Space Research and Technology（国家空间研究与技术中心）；Nazarbayev University（纳扎尔巴耶夫大学）；Purdue University Fort Wayne（普渡大学富尔顿分校）
We develop a higher order generalization of the LQ decomposition and show that this decomposition plays an important role in likelihood-based estimation and testing for separable, or Kronecker structured, covariance models, such as the multilinear normal model. This role is analogous to that of the LQ decomposition in likelihood inference for the multivariate normal model. Additionally, this higher order LQ decomposition can be used to construct an alternative version of the popular higher order singular value decomposition for tensor-valued data. We also develop a novel generalization of the …
In this paper we consider the task of estimating the non-zero pattern of the sparse inverse covariance matrix of a zero-mean Gaussian random vector from a set of iid samples. Note that this is also equivalent to recovering the underlying graph structure of a sparse Gaussian Markov Random Field (GMRF). We present two novel greedy approaches to solving this problem. The first estimates the non-zero covariates of the overall inverse covariance matrix using a series of global forward and backward greedy steps. The second estimates the neighborhood of each node in the graph separately, again using …
