Implicit probabilistic models are models defined naturally in terms of a sampling procedure and often induces a likelihood function that cannot be expressed explicitly. We develop a simple method for estimating parameters in implicit models that does not require knowledge of the form of the likelihood function or any derived quantities, but can be shown to be equivalent to maximizing likelihood under some conditions. Our result holds in the non-asymptotic parametric setting, where both the capacity of the model and the number of data examples are finite. We also demonstrate encouraging experimental results.
University of California, Berkeley（加州大学伯克利分校）；University of Toronto（多伦多大学）
Single-image super-resolution (SISR) is a canonical problem with diverse applications. Leading methods like SRGAN produce images that contain various artifacts, such as high-frequency noise, hallucinated colours and shape distortions, which adversely affect the realism of the result. In this paper, we propose an alternative approach based on an extension of the method of Implicit Maximum Likelihood Estimation (IMLE). We demonstrate greater effectiveness at noise reduction and preservation of the original colours and shapes, yielding more realistic super-resolved images.
Mila（米拉人工智能研究所）；Université de Montréal（蒙特利尔大学）；Vector Institute（向量研究所）；University of Toronto（多伦多大学）；Google Research（谷歌研究院）；Facebook（脸书）
The shortcomings of maximum likelihood estimation in the context of model-based reinforcement learning have been highlighted by an increasing number of papers. When the model class is misspecified or has a limited representational capacity, model parameters with high likelihood might not necessarily result in high performance of the agent on a downstream control task. To alleviate this problem, we propose an end-to-end approach for model learning which directly optimizes the expected returns using implicit differentiation. We treat a value function that satisfies the Bellman optimality operato…
University of Toulouse（图卢兹大学）；Institut Universitaire de France（法国大学研究院）；Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）；Centrale Lille（里尔中央理工学院）
Despite their advantages, normalizing flows generally suffer from several shortcomings including their tendency to generate unrealistic data (e.g., images) and their failing to detect out-of-distribution data. One reason for these deficiencies lies in the training strategy which traditionally exploits a maximum likelihood principle only. This paper proposes a new training paradigm based on a hybrid objective function combining the maximum likelihood principle (MLE) and a sliced-Wasserstein distance. Results obtained on synthetic toy examples and real image data sets show better generative abil…
The complexity of a maximum likelihood estimation is measured by its maximum likelihood degree ($ML$ degree). In this paper we study the maximum likelihood problem associated to chemical networks composed by one single chemical reaction under the equilibrium assumption.
Strong consistency of the quasi-maximum likelihood estimator is given for a general class of multidimensional causal processes based on asyMmetric laplacian innovation.
Maximum likelihood estimation is a fundamental optimization problem in statistics. We study this problem on manifolds of matrices with bounded rank. These represent mixtures of distributions of two independent discrete random variables. We determine the maximum likelihood degree for a range of determinantal varieties, and we apply numerical algebraic geometry to compute all critical points of their likelihood functions. This led to the discovery of maximum likelihood duality between matrices of complementary ranks, a result proved subsequently by Draisma and Rodriguez.
We have shown in previous work that statistical inference for cooperative sequential adsorption model can be based on maximum likelihood estimation. In this paper we continue this research and establish asymptotic normality of the maximum likelihood estimator in thermodynamic limit. We also perform and discuss some numerical simulations of the model.
We show that algebraic varieties with maximum likelihood degree one are exactly the images of reduced A-discriminantal varieties under monomial maps with finite fibers. The maximum likelihood estimator corresponding to such a variety is Kapranov's Horn uniformization. This extends Kapranov's characterization of A-discriminantal hypersurfaces to varieties of arbitrary codimension.
Spatial-temporal linear model and the corresponding likelihood-based statistical inference are important tools for the analysis of spatial-temporal lattice data. In this paper, we study the asymptotic properties of maximum likelihood estimates under a general asymptotic framework for spatial-temporal linear models. We propose mild regularity conditions on the spatial-temporal weight matrices and derive the asymptotic properties (consistency and asymptotic normality) of maximum likelihood estimates. A simulation study is conducted to examine the finite-sample properties of the maximum likelihoo…
This paper considers the asymptotic properties of the recursive maximum likelihood estimation in hidden Markov models. The paper is focused on the asymptotic behavior of the log-likelihood function and on the point-convergence and convergence rate of the recursive maximum likelihood estimator. Using the principle of analytical continuation, the analyticity of the asymptotic log-likelihood function is shown for analytically parameterized hidden Markov models. Relying on this fact and some results from differential geometry (Lojasiewicz inequality), the almost sure point-convergence of the recur…
We propose a novel molecular computing scheme for statistical inference. We focus on the much-studied statistical inference problem of computing maximum likelihood estimators for log-linear models. Our scheme takes log-linear models to reaction systems, and the observed data to initial conditions, so that the corresponding equilibrium of each reaction system encodes the corresponding maximum likelihood estimator. The main idea is to exploit the coincidence between thermodynamic entropy and statistical entropy. We map a Maximum Entropy characterization of the maximum likelihood estimator onto a…
We investigate the asymptotic properties of maximum likelihood estimators of the drift parameter for fractional vasicek model driven by a sub-fractional Brownian motion.
In this paper, we present a maximum likelihood method for estimating the parameters of a univariate Hawkes process with self-excitation or inhibition. Our work generalizes techniques and results that were restricted to the self-exciting scenario. The proposed estimator is implemented for the classical exponential kernel and we show that, in the inhibition context, our procedure provides more accurate estimations than current alternative approaches.
Maximum likelihood estimation has been extensively used in the joint analysis of repeated measurements and survival time. However, there is a lack of theoretical justification of the asymptotic properties for the maximum likelihood estimators. This paper intends to fill this gap. Specifically, we prove the consistency of the maximum likelihood estimators and derive their asymptotic distributions. The maximum likelihood estimators are shown to be semiparametrically efficient.
We establish connections between invariant theory and maximum likelihood estimation for discrete statistical models. We show that norm minimization over a torus orbit is equivalent to maximum likelihood estimation in log-linear models. We use notions of stability under a torus action to characterize the existence of the maximum likelihood estimate, and discuss connections to scaling algorithms.
Maximum likelihood estimation is a fundamental computational problem in statistics. In this note, we give a bound for the maximum likelihood degree of algebraic statistical models for discrete data. As usual, such models are identified with special very affine varieties. Using earlier work of Franecki and Kapranov, we prove that the maximum likelihood degree is always less or equal to the signed intersection-cohomology Euler characteristic. We construct counterexamples to a bound in terms of the usual Euler characteristic conjectured by Huh and Sturmfels.
We consider the problem of estimating the distribution function, the density and the hazard rate of the (unobservable) event time in the current status model. A well studied and natural nonparametric estimator for the distribution function in this model is the nonparametric maximum likelihood estimator (MLE). We study two alternative methods for the estimation of the distribution function, assuming some smoothness of the event time distribution. The first estimator is based on a maximum smoothed likelihood approach. The second method is based on smoothing the (discrete) MLE of the distribution…
In this paper the maximum likelihood equations for the parameters of the Weight Lindley distribution are studied considering different types of censoring, such as, type I, type II and random censoring mechanism. A numerical simulation study is perform to evaluate the maximum likelihood estimates. The proposed methodology is illustrated in a real data set.
Upper and lower class functions for the maximum likelihood estimator of the arrival and the service rates in a $GI/G/1$ queue are studied and the results are verified for M/M/1 queue.
In this paper, using spectral theory of Hilbertian operators, we study ARMA Gaussian processes indexed by graphs. We extend Whittle maximum likelihood estimation of the parameters for the corresponding spectral density and show their asymptotic optimality.
Empirical economic research frequently applies maximum likelihood estimation in cases where the likelihood function is analytically intractable. Most of the theoretical literature focuses on maximum simulated likelihood (MSL) estimators, while empirical and simulation analyzes often find that alternative approximation methods such as quasi-Monte Carlo simulation, Gaussian quadrature, and integration on sparse grids behave considerably better numerically. This paper generalizes the theoretical results widely known for MSL estimators to a general set of maximum approximated likelihood (MAL) esti…
In finite mixtures of location-scale distributions, if there is no constraint or penalty on the parameters, then the maximum likelihood estimator does not exist because the likelihood is unbounded. To avoid this problem, we consider a penalized likelihood, where the penalty is a function of the minimum of the ratios of the scale parameters and the sample size. It is shown that the penalized maximum likelihood estimator is strongly consistent. We also analyze the consistency of a penalized maximum likelihood estimator where the penalty is imposed on the scale parameters themselves.
The von Mises-Fisher distribution is one of the most widely used probability distributions to describe directional data. Finite mixtures of von Mises-Fisher distributions have found numerous applications. However, the likelihood function for the finite mixture of von Mises-Fisher distributions is unbounded and consequently the maximum likelihood estimation is not well defined. To address the problem of likelihood degeneracy, we consider a penalized maximum likelihood approach whereby a penalty function is incorporated. We prove strong consistency of the resulting estimator. An Expectation-Maxi…
Multiple root estimation problems in statistical inference arise in many contexts in the literature. In the context of maximum likelihood estimation, the existence of multiple roots causes uncertainty in the computation of maximum likelihood estimators using hill-climbing algorithms, and consequent difficulties in the resulting statistical inference. In this paper, we study the multiple roots phenomenon in maximum likelihood estimation for factor analysis. We prove that the corresponding likelihood equations have uncountably many feasible solutions even in the simplest cases. For the case in w…
In this paper, insight is given in the techniques used to compute asymptotic expansions. In a broad fashion the technique is described. Most of the results apply to the paper "An expansion for the maximum likelihood estimator and its distribution function", which will be submitted.
In this article, we investigate Maximum Likelihood Estimation with tools from Tropical Geometry and Bernstein--Sato theory. We investigate the critical points of very affine varieties and study their asymptotic behavior. We relate these asymptotics to particular rays in the tropical variety as well as to Bernstein--Sato ideals and give a connection to Maximum Likelihood Estimation in Statistics.
We discuss maximum likelihood estimation of parameters for models governed by a stochastic differential equation driven by a mixed fractional Brownian motion with random effects.
We give a statistical interpretation of entropic optimal transport by showing that performing maximum-likelihood estimation for Gaussian deconvolution corresponds to calculating a projection with respect to the entropic optimal transport distance. This structural result gives theoretical support for the wide adoption of these tools in the machine learning community.
This paper deals with the problems of consistence and strong consistence of the maximum likelihood estimators of the mean and variance of the drift fractional Brownian motions observed at discrete time instants. A central limit theorem for these estimators is also obtained by using the Malliavin calculus.
We characterize the existence of the maximum likelihood estimator for discrete exponential families. Our criterion is simple to apply as we show in various settings, most notably for exponential models of random graphs. As an application, we point out the size of independent identically distributed samples for which the maximum likelihood estimator exists with high probability.
We consider covariance parameter estimation for a Gaussian process under inequality constraints (boundedness, monotonicity or convexity) in fixed-domain asymptotics. We address the estimation of the variance parameter and the estimation of the microergodic parameter of the Matérn and Wendland covariance functions. First, we show that the (unconstrained) maximum likelihood estimator has the same asymptotic distribution, unconditionally and conditionally to the fact that the Gaussian process satisfies the inequality constraints. Then, we study the recently suggested constrained maximum likelihoo…
This paper deals with multivariate Gaussian models for which the covariance matrix is a Kronecker product of two matrices. We consider maximum likelihood estimation of the model parameters, in particular of the covariance matrix. There is no explicit expression for the maximum likelihood estimator of a Kronecker product covariance matrix. The main question in this paper is whether the maximum likelihood estimator of the covariance matrix exists and if it is unique. The answers are different for different models that we consider.
We establish the consistency of a nonparametric maximum likelihood estimator for a class of stochastic inverse problems. We proceed by embedding the framework into the general settings of early results of Pfanzagl related to mixtures.
We consider maximum likelihood estimation with data from a bivariate Gaussian process with a separable exponential covariance model under fixed domain asymptotic. We first characterize the equivalence of Gaussian measures under this model. Then consistency and asymptotic distribution for the microergodic parameters are established. A simulation study is presented in order to compare the finite sample behavior of the maximum likelihood estimator with the given asymptotic distribution.
We describe Monte Carlo approximation to the maximum likelihood estimator in models with intractable norming constants and explanatory variables. We consider both sources of randomness (due to the initial sample and to Monte Carlo simulations) and prove asymptotical normality of the estimator.
In this paper, we study sample size thresholds for maximum likelihood estimation for tensor normal models. Given the model parameters and the number of samples, we determine whether, almost surely, (1) the likelihood function is bounded from above, (2) maximum likelihood estimates (MLEs) exist, and (3) MLEs exist uniquely. We obtain a complete answer for both real and complex models. One consequence of our results is that almost sure boundedness of the log-likelihood function guarantees almost sure existence of an MLE. Our techniques are based on invariant theory and castling transforms.
We study the critical points of the likelihood function over the Fermat hypersurface. This problem is related to one of the main problems in statistical optimization: maximum likelihood estimation. The number of critical points over a projective variety is a topological invariant of the variety and is called maximum likelihood degree. We provide closed formulas for the maximum likelihood degree of any Fermat curve in the projective plane and of Fermat hypersurfaces of degree 2 in any projective space. Algorithmic methods to compute the ML degree of a generic Fermat hypersurface are developed t…
Maximum likelihood estimators are proposed for the parameters and the densities in a semiparametric density ratio model in which the nonparametric baseline density is approximated by the Bernstein polynomial model. The EM algorithm is used to obtain the maximum approximate Bernstein likelihood estimates. Simulation study shows that the performance of the proposed method is much better than the existing ones. The proposed method is illustrated by real data examples. Some asymptotic results are also presented and proved.
This paper gives a new approach for the maximum likelihood estimation of the joint of the location and scale of the Cauchy distribution. We regard the joint as a single complex parameter and derive a new form of the likelihood equation of a complex variable. Based on the equation, we provide a new iterative scheme approximating the maximum likelihood estimate. We also handle the equation in an algebraic manner and derive a polynomial containing the maximum likelihood estimate as a root. This algebraic approach provides another scheme approximating the maximum likelihood estimate by root-findin…
Maximum likelihood estimation is a common method of estimating the parameters of the probability distribution from a given sample. This paper aims to introduce the maximum likelihood estimation in the framework of sublinear expectation. We find the maximum likelihood estimator for the parameters of the maximal distribution via the solution of the associated minimax problem, which coincides with the optimal unbiased estimation given by Jin and Peng \cite{JP21}. A general estimation method for samples with dependent structure is also provided. This result provides a theoretical foundation for th…
We introduce the package "GraphicalModelsMLE" for computing the maximum likelihood estimates (MLEs) of a Gaussian graphical model in the computer algebra system Macaulay2. This package allows the computation of MLEs for the class of loopless mixed graphs. Additional functionality allows the user to explore the underlying algebraic structure of the model, such as its maximum likelihood degree and the ideal of score equations.
Invertibility conditions for observation-driven time series models often fail to be guaranteed in empirical applications. As a result, the asymptotic theory of maximum likelihood and quasi-maximum likelihood estimators may be compromised. We derive considerably weaker conditions that can be used in practice to ensure the consistency of the maximum likelihood estimator for a wide class of observation-driven time series models. Our consistency results hold for both correctly specified and misspecified models. The practical relevance of the theory is highlighted in a set of empirical examples. We…
We consider the asymptotic consistency of maximum likelihood parameter estimation for dynamical systems observed with noise. Under suitable conditions on the dynamical systems and the observations, we show that maximum likelihood parameter estimation is consistent. Our proof involves ideas from both information theory and dynamical systems. Furthermore, we show how some well-studied properties of dynamical systems imply the general statistical properties related to maximum likelihood estimation. Finally, we exhibit classical families of dynamical systems for which maximum likelihood estimation…
The maximum likelihood principle is widely used in statistics, and the associated estimators often display good properties. indeed maximum likelihood estimators are guaranteed to be asymptotically efficient under mild conditions. However in some settings, one has too few samples to get a good estimation. It then becomes desirable to take into account prior information about the distribution one wants to estimate. One possible approach is to extend the maximum likelihood principle in a bayesian context, which then becomes a maximum a posteriori estimate; however this requires a distribution mod…
In this paper, expansions for the maximum likelihood estimator of location and its distribution funtion are extended to fifth order. Since the proofs are straightforward extentions of proofs given in earlier papers for orders less than the fifth, they are not given here. The purpose of the paper is mainly to present the higher order expansions.
The asymptotic variance of the maximum likelihood estimate is proved to decrease when the maximization is restricted to a subspace that contains the true parameter value. Maximum likelihood estimation allows a systematic fitting of covariance models to the sample, which is important in data assimilation. The hierarchical maximum likelihood approach is applied to the spectral diagonal covariance model with different parameterizations of eigenvalue decay, and to the sparse inverse covariance model with specified parameter values on different sets of nonzero entries. It is shown computationally t…
In this note, we propose a new approach for the proof of the consistency and normality of the maximum likelihood estimator for nonlinear AR processes with markov-switching under the assumptions of uniform exponential forgetting of the prediction filter and $α$-mixing property. We show that in the linear and Gaussian case our assumptions are fully satisfied.
This paper deals with a general class of observation-driven time series models with a special focus on time series of counts. We provide conditions under which there exist strict-sense stationary and ergodic versions of such processes. The consistency of the maximum likelihood estimators is then derived for well- specified and misspecified models.
The log-concave maximum likelihood estimator (MLE) problem answers: for a set of points $X_1,...X_n \in \mathbb R^d$, which log-concave density maximizes their likelihood? We present a characterization of the log-concave MLE that leads to an algorithm with runtime $poly(n,d, \frac 1 ε,r)$ to compute a log-concave distribution whose log-likelihood is at most $ε$ less than that of the MLE, and $r$ is parameter of the problem that is bounded by the $\ell_2$ norm of the vector of log-likelihoods the MLE evaluated at $X_1,...,X_n$.
The maximum likelihood estimation of the left-truncated log-logistic distribution with a given truncation point is analyzed in detail from both mathematical and numerical perspectives. These maximum likelihood equations often do not possess a solution, even for small truncations. A simple criterion is provided for the existence of a regular maximum likelihood solution. In this case a profile likelihood function can be constructed and the optimisation problem is reduced to one dimension. When the maximum likelihood equations do not admit a solution for certain data samples, it is shown that the…
My first paper exclusively about ridge regression was published in Technometrics and chosen for invited presentation at the 1975 Joint Statistical Meetings in Atlanta. Unfortunately, that paper contained a wide range of assorted details and results. Luckily, Gary McDonald's published discussion of that paper focused primarily on my use of Maximum Likelihood estimation under normal distribution-theory. In this review of some results from all four of my ridge publications between 1975 and 2022, I highlight the Maximum Likelihood findings that appear to be most important in practical application …
Maximum likelihood estimation in statistics leads to the problem of maximizing a product of powers of polynomials. We study the algebraic degree of the critical equations of this optimization problem. This degree is related to the number of bounded regions in the corresponding arrangement of hypersurfaces, and to the Euler characteristic of the complexified complement. Under suitable hypotheses, the maximum likelihood degree equals the top Chern class of a sheaf of logarithmic differential forms. Exact formulae in terms of degrees and Newton polytopes are given for polynomials with generic coe…
We study the problem of maximum likelihood estimation for $3$-dimensional linear spaces of $3\times 3$ symmetric matrices from the point of view of algebraic statistics where we view these nets of conics as linear concentration or linear covariance models of Gaussian distributions on $\mathbb{R}^3$. In particular, we study the reciprocal surfaces of nets of conics which are rational surfaces in $\mathbb{P}^5$. We show that the reciprocal surfaces are projections from the Veronese surface and determine their intersection with the polar nets. This geometry explains the maximum likelihood degrees…
