Different change-point type models encountered in statistical inference for stochastic processes give rise to different limiting likelihood ratio processes. In this paper we consider two such likelihood ratios. The first one is an exponential functional of a two-sided Poisson process driven by some parameter, while the second one is an exponential functional of a two-sided Brownian motion. We establish that for sufficiently small values of the parameter, the Poisson type likelihood ratio can be approximated by the Brownian type one. As a consequence, several statistically interesting quantities (such as limiting variances of different estimators) related to the first likelihood ratio can also be approximated by those related to the second one. Finally, we discuss the asymptotics of the large values of the parameter and illustrate the results by numerical simulations.
Different change-point type models encountered in statistical inference for stochastic processes give rise to different limiting likelihood ratio processes. In a previous paper of one of the authors it was established that one of these likelihood ratios, which is an exponential functional of a two-sided Poisson process driven by some parameter, can be approximated (for sufficiently small values of the parameter) by another one, which is an exponential functional of a two-sided Brownian motion. In this paper we consider yet another likelihood ratio, which is the exponent of a two-sided compound…
Consider the likelihood ratio test (LRT) statistics for the independence of sub-vectors from a $p$-variate normal random vector. We are devoted to deriving the limiting distributions of the LRT statistics based on a random sample of size $n$. It is well known that the limit is chi-square distribution when the dimension of the data or the number of the parameters are fixed. In a recent work by Qi, Wang and Zhang (Ann Inst Stat Math (2019) 71: 911--946), it was shown that the LRT statistics are asymptotically normal under condition that the lengths of the normal random sub-vectors are relatively…
There is a difficulty in finding an estimate of variance of the profile likelihood estimator in the joint model of longitudinal and survival data. We solve the difficulty by introducing the ``statistical generalized derivative''. The derivative is used to show the asymptotic normality of the estimator without assuming the second derivative of the density function in the model exists.
We extend the correspondence between two-stage coding procedures in data compression and penalized likelihood procedures in statistical estimation. Traditionally, this had required restriction to countable parameter spaces. We show how to extend this correspondence in the uncountable parameter case. Leveraging the description length interpretations of penalized likelihood procedures we devise new techniques to derive adaptive risk bounds of such procedures. We show that the existence of certain countable coverings of the parameter space implies adaptive risk bounds and thus our theory is quite…
We consider maximum likelihood estimation for both causal and noncausal autoregressive time series processes with non-Gaussian $α$-stable noise. A nondegenerate limiting distribution is given for maximum likelihood estimators of the parameters of the autoregressive model equation and the parameters of the stable noise distribution. The estimators for the autoregressive parameters are $n^{1/α}$-consistent and converge in distribution to the maximizer of a random function. The form of this limiting distribution is intractable, but the shape of the distribution for these estimators can be examine…
Maximum-likelihood estimation (MLE) is arguably the most important tool for statisticians, and many methods have been developed to find the MLE. We present a new inequality involving posterior distributions of a latent variable that holds under very general conditions. It is related to the EM algorithm and has a clear potential for being used in a similar fashion.
Statistical models and methods for determinantal point processes (DPPs) seem largely unexplored. We demonstrate that DPPs provide useful models for the description of spatial point pattern datasets where nearby points repel each other. Such data are usually modelled by Gibbs point processes, where the likelihood and moment expressions are intractable and simulations are time consuming. We exploit the appealing probabilistic properties of DPPs to develop parametric models, where the likelihood and moment expressions can be easily evaluated and realizations can be quickly simulated. We discuss h…
The stratified proportional intensity model generalizes Cox's proportional intensity model by allowing different groups of the population under study to have distinct baseline intensity functions. In this article, we consider the problem of estimation in this model when the variable indicating the stratum is unobserved for some individuals in the studied sample. In this setting, we construct nonparametric maximum likelihood estimators for the parameters of the stratified model and we establish their consistency and asymptotic normality. Consistent estimators for the limiting variances are also…
Multiple root estimation problems in statistical inference arise in many contexts in the literature. In the context of maximum likelihood estimation, the existence of multiple roots causes uncertainty in the computation of maximum likelihood estimators using hill-climbing algorithms, and consequent difficulties in the resulting statistical inference. In this paper, we study the multiple roots phenomenon in maximum likelihood estimation for factor analysis. We prove that the corresponding likelihood equations have uncountably many feasible solutions even in the simplest cases. For the case in w…
A discrete statistical model is a subset of a probability simplex. Its maximum likelihood estimator (MLE) is a retraction from that simplex onto the model. We characterize all models for which this retraction is a rational function. This is a contribution via real algebraic geometry which rests on results due to Huh and Kapranov on Horn uniformization. We present an algorithm for constructing models with rational MLE, and we demonstrate it on a range of instances. Our focus lies on models familiar to statisticians, like Bayesian networks, decomposable graphical models, and staged trees.
The asymptotic normality of the Maximum Likelihood Estimator (MLE) is a cornerstone of statistical theory. In the present paper, we provide sharp explicit upper bounds on Zolotarev-type distances between the exact, unknown distribution of the MLE and its limiting normal distribution. Our approach to this fundamental issue is based on a sound combination of the Delta method, Stein's method, Taylor expansions and conditional expectations, for the classical situations where the MLE can be expressed as a function of a sum of independent and identically distributed terms. This encompasses in partic…
In this paper, we propose a new statistical inference method for massive data sets, which is very simple and efficient by combining divide-and-conquer method and empirical likelihood. Compared with two popular methods (the bag of little bootstrap and the subsampled double bootstrap), we make full use of data sets, and reduce the computation burden. Extensive numerical studies and real data analysis demonstrate the effectiveness and flexibility of our proposed method. Furthermore, the asymptotic property of our method is derived.
Statistical inference for discrete time observations of an affine stochastic delay differential equation is considered. The main focus is on maximum pseudo-likelihood estimators, which are easy to calculate in practice. A more general class of prediction-based estimating functions is investigated as well. In particular, the optimal prediction-based estimating function and the asymptotic properties of the estimators are derived. The maximum pseudo-likelihood estimator is a particular case, and an expression is found for the efficiency loss when using the maximum pseudo-likelihood estimator, rat…
This paper deals with the estimation problem of misspecified ergodic Lévy driven stochastic differential equation models based on high-frequency samples. We utilize the widely applicable and tractable Gaussian quasi-likelihood approach which focuses on (conditional) mean and variance structure. It is shown that the corresponding Gaussian quasi-likelihood estimators of drift and scale parameters satisfy tail probability estimates and asymptotic normality at the same rate as correctly specified case. In this process, extended Poisson equation for time-homogeneous Feller Markov processes plays an…
The law of likelihood underlies a general framework, known as the likelihood paradigm, for representing and interpreting statistical evidence. As stated, the law applies only to simple hypotheses, and there have been reservations about extending the law to composite hypotheses, despite their tremendous relevance in statistical applications. This paper proposes a generalization of the law of likelihood for composite hypotheses. The generalized law is developed in an axiomatic fashion, illustrated with real examples, and examined in an asymptotic analysis. Previous concerns about including compo…
This paper discusses the connection between mathematical finance and statistical modelling which turns out to be more than a formal mathematical correspondence. We like to figure out how common results and notions in statistics and their meaning can be translated to the world of mathematical finance and vice versa. A lot of similarities can be expressed in terms of LeCam's theory for statistical experiments which is the theory of the behaviour of likelihood processes. For positive prices the arbitrage free financial assets fit into filtered experiments. It is shown that they are given by filte…
We give a statistical interpretation of entropic optimal transport by showing that performing maximum-likelihood estimation for Gaussian deconvolution corresponds to calculating a projection with respect to the entropic optimal transport distance. This structural result gives theoretical support for the wide adoption of these tools in the machine learning community.
Likelihood-based methods of statistical inference provide a useful general methodology that is appealing, as a straightforward asymptotic theory can be applied for their implementation. It is important to assess the relationships between different likelihood-based inferential procedures in terms of accuracy and adherence to key principles of statistical inference, in particular those relating to conditioning on relevant ancillary statistics. An analysis is given of the stability properties of a general class of likelihood-based statistics, including those derived from forms of adjusted profile…
For random samples of size n obtained from p-variate normal distributions, we consider the classical likelihood ratio tests (LRT) for their means and covariance matrices in the high-dimensional setting. These test statistics have been extensively studied in multivariate analysis and their limiting distributions under the null hypothesis were proved to be chi-square distributions as n goes to infinity and p remains fixed. In this paper, we consider the high-dimensional case where both p and n go to infinity with p=n/y in (0, 1]. We prove that the likelihood ratio test statistics under this assu…
The performance of multiple hypothesis testing is known to be affected by the statistical dependence among random variables involved. The mechanisms responsible for this, however, are not well understood. We study the effects of the dependence structure of a finite state hidden Markov model (HMM) on the likelihood ratios critical for optimal multiple testing on the hidden states. Various convergence results are obtained for the likelihood ratios as the observations of the HMM form an increasing long chain. Analytic expansions of the first and second order derivatives are obtained for the case …
Fisher's Method of Maximum Likelihood is shown to be a procedure for the construction of likelihood intervals or regions, instead of a procedure of point estimation. Based on Fisher's articles and books it is justified that by estimation Fisher meant the construction of likelihood intervals or regions from appropriate likelihood function and that an estimate is a statistic, that is, a function from a sample space to a parameter space such that the likelihood function obtained from the sampling distribution of the statistic at the observed value of the statistic is used to construct likelihood …
In this paper, we develop asymptotic theories for a class of latent variable models for large-scale multi-relational networks. In particular, we establish consistency results and asymptotic error bounds for the (penalized) maximum likelihood estimators when the size of the network tends to infinity. The basic technique is to develop a non-asymptotic error bound for the maximum likelihood estimators through large deviations analysis of random fields. We also show that these estimators are nearly optimal in terms of minimax risk.
This is an invited contribution to the discussion on Professor Deborah Mayo's paper, "On the Birnbaum argument for the strong likelihood principle," to appear in Statistical Science. Mayo clearly demonstrates that statistical methods violating the likelihood principle need not violate either the sufficiency or conditionality principle, thus refuting Birnbaum's claim. With the constraints of Birnbaum's theorem lifted, we revisit the foundations of statistical inference, focusing on some new foundational principles, the inferential model framework, and connections with sufficiency and conditioni…
We propose a novel molecular computing scheme for statistical inference. We focus on the much-studied statistical inference problem of computing maximum likelihood estimators for log-linear models. Our scheme takes log-linear models to reaction systems, and the observed data to initial conditions, so that the corresponding equilibrium of each reaction system encodes the corresponding maximum likelihood estimator. The main idea is to exploit the coincidence between thermodynamic entropy and statistical entropy. We map a Maximum Entropy characterization of the maximum likelihood estimator onto a…
In some non-regular statistical estimation problems, the limiting likelihood processes are functionals of fractional Brownian motion (fBm) with Hurst's parameter H; 0 < H <=? 1. In this paper we present several analytical and numerical results on the moments of Pitman estimators represented in the form of integral functionals of fBm. We also provide Monte Carlo simulation results for variances of Pitman and asymptotic maximum likelihood estimators.
We study the empirical likelihood approach to construct confidence intervals for the optimal value and the optimality gap of a given solution, henceforth quantify the statistical uncertainty of sample average approximation, for optimization problems with expected value objectives and constraints where the underlying probability distributions are observed via limited data. This approach relies on two distributionally robust optimization problems posited over the uncertain distribution, with a divergence-based uncertainty set that is suitably calibrated to provide asymptotic statistical guarante…
Spatial-temporal linear model and the corresponding likelihood-based statistical inference are important tools for the analysis of spatial-temporal lattice data. In this paper, we study the asymptotic properties of maximum likelihood estimates under a general asymptotic framework for spatial-temporal linear models. We propose mild regularity conditions on the spatial-temporal weight matrices and derive the asymptotic properties (consistency and asymptotic normality) of maximum likelihood estimates. A simulation study is conducted to examine the finite-sample properties of the maximum likelihoo…
In many complex statistical models maximum likelihood estimators cannot be calculated. In the paper we solve this problem using Markov chain Monte Carlo approximation of the true likelihood. In the main result we prove asymptotic normality of the estimator, when both sample sizes (the initial and Monte Carlo one) tend to infinity. Our result can be applied to models with intractable norming constants and missing data models.
Community detection is one of the most important problems in network analysis. Among many algorithms proposed for this task, methods based on statistical inference are of particular interest: they are mathematically sound and were shown to provide partitions of good quality. Statistical inference methods are based on fitting some random graph model (a.k.a. null model) to the observed network by maximizing the likelihood. The choice of this model is extremely important and is the main focus of the current study. We provide an extensive theoretical and empirical analysis to compare several model…
Statistical models with latent structure have a history going back to the 1950s and have seen widespread use in the social sciences and, more recently, in computational biology and in machine learning. Here we study the basic latent class model proposed originally by the sociologist Paul F. Lazarfeld for categorical variables, and we explain its geometric structure. We draw parallels between the statistical and geometric properties of latent class models and we illustrate geometrically the causes of many problems associated with maximum likelihood estimation and related statistical inference. …
This article focuses on the study of toric algebraic statistical models which correspond to toric Del Pezzo surfaces with Du Val singularities. A closed-form for the Maximum Likelihood Estimate of algebraic statistical models which correspond to cubic and quartic toric Del Pezzo surfaces with Du Val singular points is given. Also, we calculate the ML degrees of some toric Del Pezzo surfaces of degree less than or equal to six, which equals the degree of the surface in all the case but one, namely the quintic with two points of type $\mathbb{A}_1$.
Parametric inference posits a statistical model that is a specified family of probability distributions. Restricted inference, e.g., restricted likelihood ratio testing, attempts to exploit the structure of a statistical submodel that is a subset of the specified family. We consider the problem of testing a simple hypothesis against alternatives from such a submodel. In the case of an unknown submodel, it is not clear how to realize the benefits of restricted inference. To do so, we first construct information tests that are locally asymptotically equivalent to likelihood ratio tests. Informat…
Multivariate normal mixtures provide a flexible model for high-dimensional data. They are widely used in statistical genetics, statistical finance, and other disciplines. Due to the unboundedness of the likelihood function, classical likelihood-based methods, which may have nice practical properties, are inconsistent. In this paper, we recommend a penalized likelihood method for estimating the mixing distribution. We show that the maximum penalized likelihood estimator is strongly consistent when the number of components has a known upper bound. We also explore a convenient EM-algorithm for co…
We have shown in previous work that statistical inference for cooperative sequential adsorption model can be based on maximum likelihood estimation. In this paper we continue this research and establish asymptotic normality of the maximum likelihood estimator in thermodynamic limit. We also perform and discuss some numerical simulations of the model.
In this paper we consider the problem of parameter estimation in the $p$-spin Curie-Weiss model, for $p \geq 3$. We provide a complete description of the limiting properties of the maximum likelihood (ML) estimates of the inverse temperature and the magnetic field given a single realization from the $p$-spin Curie-Weiss model, complementing the well-known results in the 2-spin case (Comets and Gidas (1991)). Our results unearth various new phase transitions and surprising limit theorems, such as the existence of a 'critical' curve in the parameter space, where the limiting distribution of the …
The likelihood function represents statistical evidence in the context of data and a probability model. Considerable theory has demonstrated that evidence strength for different parameter values can be interpreted from the ratio of likelihoods at different points on the likelihood curve. The likelihood function can, however, be unknown or difficult to compute; e.g. for genetic association studies with a binary outcome in large multi-generational families. Composite likelihood is a convenient alternative to using the real likelihood and here we show composite likelihoods have valid evidential i…
Neural networks (NN) play a central role in modern Artificial intelligence (AI) technology and has been successfully used in areas such as natural language processing and image recognition. While majority of NN applications focus on prediction and classification, there are increasing interests in studying statistical inference of neural networks. The study of NN statistical inference can enhance our understanding of NN statistical proprieties. Moreover, it can facilitate the NN-based hypothesis testing that can be applied to hypothesis-driven clinical and biomedical research. In this paper, we…
Empirical likelihood approach is one of non-parametric statistical methods, which is applied to the hypothesis testing or construction of confidence regions for pivotal unknown quantities. This method has been applied to the case of independent identically distributed random variables and second order stationary processes. In recent years, we observe heavy-tailed data in many fields. To model such data suitably, we consider symmetric scalar and multivariate $α$-stable linear processes generated by infinite variance innovation sequence. We use a Whittle likelihood type estimating function in th…
We consider statistical models driven by Gaussian and non-Gaussian self-similar processes with long memory and we construct maximum likelihood estimators (MLE) for the drift parameter. Our approach is based on the approximation by random walks of the driving noise. We study the asymptotic behavior of the estimators and we give some numerical simulations to illustrate our results.
We study the problem of maximum likelihood estimation for general patterns of bivariate missing data for normal and multinomial random variables, under the assumption that the data is missing at random (MAR). For normal data, the score equations have nine complex solutions, at least one of which is real and statistically significant. Our computations suggest that the number of real solutions is related to whether or not the MAR assumption is satisfied. In the multinomial case, all solutions to the score equations are real and the number of real solutions grows exponentially in the number of st…
We establish connections between invariant theory and maximum likelihood estimation for discrete statistical models. We show that norm minimization over a torus orbit is equivalent to maximum likelihood estimation in log-linear models. We use notions of stability under a torus action to characterize the existence of the maximum likelihood estimate, and discuss connections to scaling algorithms.
Maximum likelihood estimation is a fundamental computational problem in statistics. In this note, we give a bound for the maximum likelihood degree of algebraic statistical models for discrete data. As usual, such models are identified with special very affine varieties. Using earlier work of Franecki and Kapranov, we prove that the maximum likelihood degree is always less or equal to the signed intersection-cohomology Euler characteristic. We construct counterexamples to a bound in terms of the usual Euler characteristic conjectured by Huh and Sturmfels.
