Invariant theory and scaling algorithms for maximum likelihood estimation
Carlos Améndola, Kathlén Kohn, Philipp Reichenbach, Anna Seigal
备注：
34 pages; The discrete part on log-linear models from version 1 is contained in the companion paper arXiv:2012.07793. (v4: very minor changes compared to v3)
We uncover connections between maximum likelihood estimation in statistics and norm minimization over a group orbit in invariant theory. We focus on Gaussian transformation families, which include matrix normal models and Gaussian graphical models given by transitive directed acyclic graphs. We use stability under group actions to characterize boundedness of the likelihood, and existence and uniqueness of the maximum likelihood estimate. Our approach reveals promising consequences of the interplay between invariant theory and statistics. In particular, existing scaling algorithms from statistics can be used in invariant theory, and vice versa.
We show that algebraic varieties with maximum likelihood degree one are exactly the images of reduced A-discriminantal varieties under monomial maps with finite fibers. The maximum likelihood estimator corresponding to such a variety is Kapranov's Horn uniformization. This extends Kapranov's characterization of A-discriminantal hypersurfaces to varieties of arbitrary codimension.
We establish connections between invariant theory and maximum likelihood estimation for discrete statistical models. We show that norm minimization over a torus orbit is equivalent to maximum likelihood estimation in log-linear models. We use notions of stability under a torus action to characterize the existence of the maximum likelihood estimate, and discuss connections to scaling algorithms.
Maximum likelihood estimation is a fundamental computational problem in statistics. In this note, we give a bound for the maximum likelihood degree of algebraic statistical models for discrete data. As usual, such models are identified with special very affine varieties. Using earlier work of Franecki and Kapranov, we prove that the maximum likelihood degree is always less or equal to the signed intersection-cohomology Euler characteristic. We construct counterexamples to a bound in terms of the usual Euler characteristic conjectured by Huh and Sturmfels.
In this article, we investigate Maximum Likelihood Estimation with tools from Tropical Geometry and Bernstein--Sato theory. We investigate the critical points of very affine varieties and study their asymptotic behavior. We relate these asymptotics to particular rays in the tropical variety as well as to Bernstein--Sato ideals and give a connection to Maximum Likelihood Estimation in Statistics.
Maximum likelihood estimation in statistics leads to the problem of maximizing a product of powers of polynomials. We study the algebraic degree of the critical equations of this optimization problem. This degree is related to the number of bounded regions in the corresponding arrangement of hypersurfaces, and to the Euler characteristic of the complexified complement. Under suitable hypotheses, the maximum likelihood degree equals the top Chern class of a sheaf of logarithmic differential forms. Exact formulae in terms of degrees and Newton polytopes are given for polynomials with generic coe…
We study the problem of maximum likelihood estimation for $3$-dimensional linear spaces of $3\times 3$ symmetric matrices from the point of view of algebraic statistics where we view these nets of conics as linear concentration or linear covariance models of Gaussian distributions on $\mathbb{R}^3$. In particular, we study the reciprocal surfaces of nets of conics which are rational surfaces in $\mathbb{P}^5$. We show that the reciprocal surfaces are projections from the Veronese surface and determine their intersection with the polar nets. This geometry explains the maximum likelihood degrees…
Maximum likelihood estimation (MLE) is a fundamental computational problem in statistics. In this paper, MLE for statistical models with discrete data is studied from an algebraic statistics viewpoint. A reformulation of the MLE problem in terms of dual varieties and conormal varieties will be given. With this description, the dual likelihood equations and the dual MLE problem are defined. We show that solving the dual MLE problem yields solutions to the MLE problem, so we can solve the MLE problem without ever determining the defining equations of the model.
The Expectation-Maximization (EM) algorithm is routinely used for the maximum likelihood estimation in the latent class analysis. However, the EM algorithm comes with no guarantees of reaching the global optimum. We study the geometry of the latent class model in order to understand the behavior of the maximum likelihood estimator. In particular, we characterize the boundary stratification of the binary latent class model with a binary hidden variable. For small models, such as for three binary observed variables, we show that this stratification allows exact computation of the maximum likelih…
We study the maximum likelihood estimation problem for several classes of toric Fano models. We start by exploring the maximum likelihood degree for all $2$-dimensional Gorenstein toric Fano varieties. We show that the ML degree is equal to the degree of the surface in every case except for the quintic del Pezzo surface with two ordinary double points and provide explicit expressions that allow one to compute the maximum likelihood estimate in closed form whenever the ML degree is less than 5. We then explore the reasons for the ML degree drop using $A$-discriminants and intersection theory. F…
Strong consistency of the quasi-maximum likelihood estimator is given for a general class of multidimensional causal processes based on asyMmetric laplacian innovation.
Spatial-temporal linear model and the corresponding likelihood-based statistical inference are important tools for the analysis of spatial-temporal lattice data. In this paper, we study the asymptotic properties of maximum likelihood estimates under a general asymptotic framework for spatial-temporal linear models. We propose mild regularity conditions on the spatial-temporal weight matrices and derive the asymptotic properties (consistency and asymptotic normality) of maximum likelihood estimates. A simulation study is conducted to examine the finite-sample properties of the maximum likelihoo…
This paper considers the asymptotic properties of the recursive maximum likelihood estimation in hidden Markov models. The paper is focused on the asymptotic behavior of the log-likelihood function and on the point-convergence and convergence rate of the recursive maximum likelihood estimator. Using the principle of analytical continuation, the analyticity of the asymptotic log-likelihood function is shown for analytically parameterized hidden Markov models. Relying on this fact and some results from differential geometry (Lojasiewicz inequality), the almost sure point-convergence of the recur…
In this paper, we study sample size thresholds for maximum likelihood estimation for tensor normal models. Given the model parameters and the number of samples, we determine whether, almost surely, (1) the likelihood function is bounded from above, (2) maximum likelihood estimates (MLEs) exist, and (3) MLEs exist uniquely. We obtain a complete answer for both real and complex models. One consequence of our results is that almost sure boundedness of the log-likelihood function guarantees almost sure existence of an MLE. Our techniques are based on invariant theory and castling transforms.
Maximum likelihood estimation has been extensively used in the joint analysis of repeated measurements and survival time. However, there is a lack of theoretical justification of the asymptotic properties for the maximum likelihood estimators. This paper intends to fill this gap. Specifically, we prove the consistency of the maximum likelihood estimators and derive their asymptotic distributions. The maximum likelihood estimators are shown to be semiparametrically efficient.
We consider the problem of estimating the distribution function, the density and the hazard rate of the (unobservable) event time in the current status model. A well studied and natural nonparametric estimator for the distribution function in this model is the nonparametric maximum likelihood estimator (MLE). We study two alternative methods for the estimation of the distribution function, assuming some smoothness of the event time distribution. The first estimator is based on a maximum smoothed likelihood approach. The second method is based on smoothing the (discrete) MLE of the distribution…
Upper and lower class functions for the maximum likelihood estimator of the arrival and the service rates in a $GI/G/1$ queue are studied and the results are verified for M/M/1 queue.
In this paper, using spectral theory of Hilbertian operators, we study ARMA Gaussian processes indexed by graphs. We extend Whittle maximum likelihood estimation of the parameters for the corresponding spectral density and show their asymptotic optimality.
Multiple root estimation problems in statistical inference arise in many contexts in the literature. In the context of maximum likelihood estimation, the existence of multiple roots causes uncertainty in the computation of maximum likelihood estimators using hill-climbing algorithms, and consequent difficulties in the resulting statistical inference. In this paper, we study the multiple roots phenomenon in maximum likelihood estimation for factor analysis. We prove that the corresponding likelihood equations have uncountably many feasible solutions even in the simplest cases. For the case in w…
We give a statistical interpretation of entropic optimal transport by showing that performing maximum-likelihood estimation for Gaussian deconvolution corresponds to calculating a projection with respect to the entropic optimal transport distance. This structural result gives theoretical support for the wide adoption of these tools in the machine learning community.
This paper deals with multivariate Gaussian models for which the covariance matrix is a Kronecker product of two matrices. We consider maximum likelihood estimation of the model parameters, in particular of the covariance matrix. There is no explicit expression for the maximum likelihood estimator of a Kronecker product covariance matrix. The main question in this paper is whether the maximum likelihood estimator of the covariance matrix exists and if it is unique. The answers are different for different models that we consider.
We establish the consistency of a nonparametric maximum likelihood estimator for a class of stochastic inverse problems. We proceed by embedding the framework into the general settings of early results of Pfanzagl related to mixtures.
We consider maximum likelihood estimation with data from a bivariate Gaussian process with a separable exponential covariance model under fixed domain asymptotic. We first characterize the equivalence of Gaussian measures under this model. Then consistency and asymptotic distribution for the microergodic parameters are established. A simulation study is presented in order to compare the finite sample behavior of the maximum likelihood estimator with the given asymptotic distribution.
In this paper, we study the log-likelihood function and Maximum Likelihood Estimate (MLE) for the matrix normal model for both real and complex models. We describe the exact number of samples needed to achieve (almost surely) three conditions, namely a bounded log-likelihood function, existence of MLEs, and uniqueness of MLEs. As a consequence, we observe that almost sure boundedness of log-likelihood function guarantees almost sure existence of an MLE, thereby proving a conjecture of Drton, Kuriki and Hoff. The main tools we use are from the theory of quiver representations, in particular, re…
This paper gives a new approach for the maximum likelihood estimation of the joint of the location and scale of the Cauchy distribution. We regard the joint as a single complex parameter and derive a new form of the likelihood equation of a complex variable. Based on the equation, we provide a new iterative scheme approximating the maximum likelihood estimate. We also handle the equation in an algebraic manner and derive a polynomial containing the maximum likelihood estimate as a root. This algebraic approach provides another scheme approximating the maximum likelihood estimate by root-findin…
The maximum likelihood principle is widely used in statistics, and the associated estimators often display good properties. indeed maximum likelihood estimators are guaranteed to be asymptotically efficient under mild conditions. However in some settings, one has too few samples to get a good estimation. It then becomes desirable to take into account prior information about the distribution one wants to estimate. One possible approach is to extend the maximum likelihood principle in a bayesian context, which then becomes a maximum a posteriori estimate; however this requires a distribution mod…
The asymptotic variance of the maximum likelihood estimate is proved to decrease when the maximization is restricted to a subspace that contains the true parameter value. Maximum likelihood estimation allows a systematic fitting of covariance models to the sample, which is important in data assimilation. The hierarchical maximum likelihood approach is applied to the spectral diagonal covariance model with different parameterizations of eigenvalue decay, and to the sparse inverse covariance model with specified parameter values on different sets of nonzero entries. It is shown computationally t…
In this note, we propose a new approach for the proof of the consistency and normality of the maximum likelihood estimator for nonlinear AR processes with markov-switching under the assumptions of uniform exponential forgetting of the prediction filter and $α$-mixing property. We show that in the linear and Gaussian case our assumptions are fully satisfied.
This paper deals with a general class of observation-driven time series models with a special focus on time series of counts. We provide conditions under which there exist strict-sense stationary and ergodic versions of such processes. The consistency of the maximum likelihood estimators is then derived for well- specified and misspecified models.
We have shown in previous work that statistical inference for cooperative sequential adsorption model can be based on maximum likelihood estimation. In this paper we continue this research and establish asymptotic normality of the maximum likelihood estimator in thermodynamic limit. We also perform and discuss some numerical simulations of the model.
The maximum likelihood degree (ML degree) measures the algebraic complexity of a fundamental optimization problem in statistics: maximum likelihood estimation. In this problem, one maximizes the likelihood function over a statistical model. The ML degree of a model is an upper bound to the number of local extrema of the likelihood function and can be expressed as a weighted sum of Euler characteristics. The independence model (i.e. rank one matrices over the probability simplex) is well known to have an ML degree of one, meaning their is a unique local maxima of the likelihood function. Howeve…
We investigate the asymptotic properties of maximum likelihood estimators of the drift parameter for fractional vasicek model driven by a sub-fractional Brownian motion.
In many complex statistical models maximum likelihood estimators cannot be calculated. In the paper we solve this problem using Markov chain Monte Carlo approximation of the true likelihood. In the main result we prove asymptotic normality of the estimator, when both sample sizes (the initial and Monte Carlo one) tend to infinity. Our result can be applied to models with intractable norming constants and missing data models.
The vanilla method in univariate extreme-value theory consists of fitting the three-parameter Generalized Extreme-Value (GEV) distribution to a sample of block maxima. Despite claims to the contrary, the asymptotic normality of the maximum likelihood estimator has never been established. In this paper, a formal proof is given using a general result on the maximum likelihood estimator for parametric families that are differentiable in quadratic mean but whose supports depend on the parameter. An interesting side result concerns the (lack of) differentiability in quadratic mean of the GEV family…
We extend the constrained maximum likelihood estimation theory for parameters of a completely identified model, proposed by Aitchison and Silvey (1958), to parameters arising from a partially identified model. With a partially identified model, some parameters of the model may only be identified through constraints imposed by additional assumptions. We show that, under certain conditions, the constrained maximum likelihood estimator exists and locally maximize the likelihood function subject to constraints. We then study the asymptotic distribution of the estimator and propose a numerical algo…
We prove the consistency and asymptotic normality of the Laplacian Quasi-Maximum Likelihood Estimator (QMLE) for a general class of causal time series including ARMA, AR($\infty$), GARCH, ARCH($\infty$), ARMA-GARCH, APARCH, ARMA-APARCH,..., processes. We notably exhibit the advantages (moment order and robustness) of this estimator compared to the classical Gaussian QMLE. Numerical simulations confirms the accuracy of this estimator.
Most statistical software packages implement numerical strategies for computation of maximum likelihood estimates in random effects models. Little is known, however, about the algebraic complexity of this problem. For the one-way layout with random effects and unbalanced group sizes, we give formulas for the algebraic degree of the likelihood equations as well as the equations for restricted maximum likelihood estimation. In particular, the latter approach is shown to be algebraically less complex. The formulas are obtained by studying a univariate rational equation whose solutions correspond …
The asymptotic normality of the Maximum Likelihood Estimator (MLE) is a long established result. Explicit bounds for the distributional distance between the distribution of the MLE and the normal distribution have recently been obtained for the case of independent random variables. In this paper, a local dependence structure is introduced between the random variables and we give upper bounds which are specified for the Wasserstein metric.
We prove the strong consistency and the asymptotic normality of the maximum likelihood estimator of the parameters of a general conditionally heteroscedastic model with $α$-stable innovations. Then, we relax the assumptions and only suppose that the innovation process converges in distribution toward a stable process. Using a pseudo maximum likelihood estimator with a stable density, we also obtain the strong consistency and the asymptotic normality of the estimator. This framework seems relevant for financial data exhibiting heavy tails. We apply this method to several financial index and com…
Empirical economic research frequently applies maximum likelihood estimation in cases where the likelihood function is analytically intractable. Most of the theoretical literature focuses on maximum simulated likelihood (MSL) estimators, while empirical and simulation analyzes often find that alternative approximation methods such as quasi-Monte Carlo simulation, Gaussian quadrature, and integration on sparse grids behave considerably better numerically. This paper generalizes the theoretical results widely known for MSL estimators to a general set of maximum approximated likelihood (MAL) esti…
In finite mixtures of location-scale distributions, if there is no constraint or penalty on the parameters, then the maximum likelihood estimator does not exist because the likelihood is unbounded. To avoid this problem, we consider a penalized likelihood, where the penalty is a function of the minimum of the ratios of the scale parameters and the sample size. It is shown that the penalized maximum likelihood estimator is strongly consistent. We also analyze the consistency of a penalized maximum likelihood estimator where the penalty is imposed on the scale parameters themselves.
We study maximum likelihood estimation in log-linear models under conditional Poisson sampling schemes. We derive necessary and sufficient conditions for existence of the maximum likelihood estimator (MLE) of the model parameters and investigate estimability of the natural and mean-value parameters under a nonexistent MLE. Our conditions focus on the role of sampling zeros in the observed table. We situate our results within the framework of extended exponential families, and we exploit the geometric properties of log-linear models. We propose algorithms for extended maximum likelihood estimat…
We discuss maximum likelihood estimation of parameters for models governed by a stochastic differential equation driven by a mixed fractional Brownian motion with random effects.
We consider covariance parameter estimation for a Gaussian process under inequality constraints (boundedness, monotonicity or convexity) in fixed-domain asymptotics. We address the estimation of the variance parameter and the estimation of the microergodic parameter of the Matérn and Wendland covariance functions. First, we show that the (unconstrained) maximum likelihood estimator has the same asymptotic distribution, unconditionally and conditionally to the fact that the Gaussian process satisfies the inequality constraints. Then, we study the recently suggested constrained maximum likelihoo…
Nonparametric identification and maximum likelihood estimation for finite-state hidden Markov models are investigated. We obtain identification of the parameters as well as the order of the Markov chain if the transition probability matrices have full-rank and are ergodic, and if the state-dependent distributions are all distinct, but not necessarily linearly independent. Based on this identification result, we develop nonparametric maximum likelihood estimation theory. First, we show that the asymptotic contrast, the Kullback--Leibler divergence of the hidden Markov model, identifies the true…
The asymptotic normality of the maximum likelihood estimator (MLE) under regularity conditions is a cornerstone of statistical theory. In this paper, we give explicit upper bounds on the distributional distance between the distribution of the MLE of a vector parameter, and the multivariate normal distribution. We work with possibly high-dimensional, independent but not necessarily identically distributed random vectors. In addition, we obtain explicit upper bounds even in cases where the MLE cannot be expressed analytically.
A hidden Markov model with trends is a hidden Markov model whose emission distributions are translated by a trend that depends on the current hidden state and on the current time. Contrary to standard hidden Markov models, such processes are not homogeneous and cannot be made homogeneous by a simple de-trending step. We show that when the trends are polynomial, the maximum likelihood estimator is able to recover the trends together with the other parameters and is strongly consistent. More precisely, the supremum norm of the difference between the true trends and the estimated ones tends to ze…
Maximum likelihood estimation is a common method of estimating the parameters of the probability distribution from a given sample. This paper aims to introduce the maximum likelihood estimation in the framework of sublinear expectation. We find the maximum likelihood estimator for the parameters of the maximal distribution via the solution of the associated minimax problem, which coincides with the optimal unbiased estimation given by Jin and Peng \cite{JP21}. A general estimation method for samples with dependent structure is also provided. This result provides a theoretical foundation for th…
Invertibility conditions for observation-driven time series models often fail to be guaranteed in empirical applications. As a result, the asymptotic theory of maximum likelihood and quasi-maximum likelihood estimators may be compromised. We derive considerably weaker conditions that can be used in practice to ensure the consistency of the maximum likelihood estimator for a wide class of observation-driven time series models. Our consistency results hold for both correctly specified and misspecified models. The practical relevance of the theory is highlighted in a set of empirical examples. We…
We consider the asymptotic consistency of maximum likelihood parameter estimation for dynamical systems observed with noise. Under suitable conditions on the dynamical systems and the observations, we show that maximum likelihood parameter estimation is consistent. Our proof involves ideas from both information theory and dynamical systems. Furthermore, we show how some well-studied properties of dynamical systems imply the general statistical properties related to maximum likelihood estimation. Finally, we exhibit classical families of dynamical systems for which maximum likelihood estimation…
Maximum-likelihood estimation (MLE) is arguably the most important tool for statisticians, and many methods have been developed to find the MLE. We present a new inequality involving posterior distributions of a latent variable that holds under very general conditions. It is related to the EM algorithm and has a clear potential for being used in a similar fashion.
