In unconstrained maximum a posteriori (MAP) and maximum likelihood estimation, the inverse of minus the merit-function Hessian matrix is an approximation of the estimate covariance matrix. In the Bayesian context of MAP estimation, it is the covariance of a normal approximation of the posterior around the mode; while in maximum likelihood estimation, it an approximation of the inverse Fisher information matrix, to which the covariance of efficient estimators converge. These measures are routinely used in system identification to evaluate the estimate uncertainties and diagnose problems such as overparametrization, improper excitation and unidentifiability. A wide variety of estimation problems in systems and control, however, can be formulated as equality-constrained optimizations with additional decision variables to exploit parallelism in computer hardware, simplify implementation and increase the convergence basin and efficiency of the nonlinear program solver. The introduction of the extra variables, however, dissociates the inverse Hessian from the covariance matrix. Instead, submatrices of the inverse Hessian of the constrained-problem's Lagrangian must be used. In this paper, we derive these relationships, showing how the estimates' covariance can be estimated directly from the augmented problem. Application examples are shown in system identification with the output-error method and joint state-path and parameter estimation.
The Gini correlation plays an important role in measuring dependence of random variables with heavy tailed distributions, whose properties are a mixture of Pearson's and Spearman's correlations. Due to the structure of this dependence measure, there are two Gini correlations between each pair of random variables, which are not equal in general. Both the Gini correlation and the equality of the two Gini correlations play important roles in Economics. In the literature, there are limited papers focusing on the inference of the Gini correlations and their equality testing. In this paper, we devel…
We propose a likelihood ratio test framework for testing normal mean vectors in high-dimensional data under two common scenarios: the one-sample test and the two-sample test with equal covariance matrices. We derive the test statistics under the assumption that the covariance matrices follow a diagonal matrix structure. In comparison with the diagonal Hotelling's tests, our proposed test statistics display some interesting characteristics. In particular, they are a summation of the log-transformed squared $t$-statistics rather than a direct summation of those components. More importantly, to d…
Empirical phi-divergence test-statistics have demostrated to be a useful technique for the simple null hypothesis to improve the finite sample behaviour of the classical likelihood ratio test-statistic, as well asfor model misspecification problems, in both cases for the one population problem. This paper introduces this methodology for two sample problems. A simulation study illustrates situations in which the new test-statistics become a competitive tool with respect to the classical z-test and the likelihood ratio test-statistic.
Penalized likelihood models are widely used to simultaneously select variables and estimate model parameters. However, the existence of weak signals can lead to inaccurate variable selection, biased parameter estimation, and invalid inference. Thus, identifying weak signals accurately and making valid inferences are crucial in penalized likelihood models. We develop a unified approach to identify weak signals and make inferences in penalized likelihood models, including the special case when the responses are categorical. To identify weak signals, we use the estimated selection probability of …
In the context of data that lie on the simplex, we investigate use of empirical and exponential empirical likelihood, and Hotelling and James statistics, to test the null hypothesis of equal population means based on two independent samples. We perform an extensive numerical study using data simulated from various distributions on the simplex. The results, taken together with practical considerations regarding implementation, support the use of bootstrap-calibrated James statistic.
Nonparametric tests for equality of multivariate distributions are frequently desired in research. It is commonly required that test-procedures based on relatively small samples of vectors accurately control the corresponding Type I Error (TIE) rates. Often, in the multivariate testing, extensions of null-distribution-free univariate methods, e.g., Kolmogorov-Smirnov and Cramer-von Mises type schemes, are not exact, since their null distributions depend on underlying data distributions. The present paper extends the density-based empirical likelihood technique in order to nonparametrically app…
In this work, we are motivated by the recent work of Zhang et al. (2019) and study a new invariant test for equality of two large scale covariance matrices. Two modified likelihood ratio tests (LRTs) by Zhang et al. (2019) are based on the sum of log of eigenvalues (or 1- eigenvalues) of the Beta-matrix. However, as the dimension increases, many eigenvalues of the Beta-matrix are close to 0 or 1 and the modified LRTs are greatly influenced by them. In this work, instead, we consider the simple sum of the eigenvalues (of the Beta-matrix) and compute its asymptotic normality when all $n_1, n_2, …
In this paper, using spectral theory of Hilbertian operators, we study ARMA Gaussian processes indexed by graphs. We extend Whittle maximum likelihood estimation of the parameters for the corresponding spectral density and show their asymptotic optimality.
In this paper, we use the method of modified signed log-likelihood ratio test for the problem of testing the equality of correlation coefficients in two independent bivariate normal distributions. We compare this method with two other %competing approaches, Fisher's Z-transform and generalized test variable, using a Monte Carlo simulation. It indicates that the proposed method is better than the other approaches, in terms of the actual sizes and powers especially when the sample sizes are unequal. We illustrate performance of the proposed approach, using a real data set.
Nonparametric identification and maximum likelihood estimation for finite-state hidden Markov models are investigated. We obtain identification of the parameters as well as the order of the Markov chain if the transition probability matrices have full-rank and are ergodic, and if the state-dependent distributions are all distinct, but not necessarily linearly independent. Based on this identification result, we develop nonparametric maximum likelihood estimation theory. First, we show that the asymptotic contrast, the Kullback--Leibler divergence of the hidden Markov model, identifies the true…
We propose a likelihood ratio statistic for forming hypothesis tests and confidence intervals for a nonparametrically estimated univariate regression function, based on the shape restriction of concavity (alternatively, convexity). Dealing with the likelihood ratio statistic requires studying an estimator satisfying a null hypothesis, that is, studying a concave least-squares estimator satisfying a further equality constraint. We study this null hypothesis least-squares estimator (NLSE) here, and use it to study our likelihood ratio statistic. The NLSE is the solution to a convex program, and …
Coordinated Science Laboratory, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校协调科学实验室）；ECE Department, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校电子与计算机工程系）
We consider the problem of identifying parameters of a particular class of Markov chains, called Bernoulli Autoregressive (BAR) processes. The structure of any BAR model is encoded by a directed graph. Incoming edges to a node in the graph indicate that the state of the node at a particular time instant is influenced by the states of the corresponding parental nodes in the previous time instant. The associated edge weights determine the corresponding level of influence from each parental node. In the simplest setup, the Bernoulli parameter of a particular node's state variable is a convex comb…
This paper considers the optimal modification of the likelihood ratio test (LRT) for the equality of two high-dimensional covariance matrices. The classical LRT is not well defined when the dimensions are larger than or equal to one of the sample sizes. In this paper, an optimally modified test that works well in cases where the dimensions may be larger than the sample sizes is proposed. In addition, the test is established under the weakest conditions on the moments and the dimensions of the samples. We also present weakly consistent estimators of the fourth moments, which are necessary for t…
We propose a new methodology for denoising, variance-stabilizing and normalizing signals whose both mean and variance are parameterized by a single unknown varying parameter, such as Poisson or scaled chi-squared. Key to our methodology is the observation that the signed and square-rooted generalized log-likelihood ratio test for the equality of the local means is approximately and asymptotically distributed as standard normal under the null. We use these test statistics within the Haar wavelet transform at each scale and location, referring to them as the likelihood ratio Haar (LRH) coefficie…
University of Bristol（布里斯托大学）；The Alan Turing Institute（艾伦·图灵研究所）；Tokyo Institute of Technology（东京工业大学）；RIKEN（理化学研究所）
Maximum Likelihood Estimators (MLE) has many good properties. For example, the asymptotic variance of MLE solution attains equality of the asymptotic Cram{é}r-Rao lower bound (efficiency bound), which is the minimum possible variance for an unbiased estimator. However, obtaining such MLE solution requires calculating the likelihood function which may not be tractable due to the normalization term of the density model. In this paper, we derive a Discriminative Likelihood Estimator (DLE) from the Kullback-Leibler divergence minimization criterion implemented via density ratio estimation and a St…
High-dimensional statistical inference with general estimating equations are challenging and remain less explored. In this paper, we study two problems in the area: confidence set estimation for multiple components of the model parameters, and model specifications test. For the first one, we propose to construct a new set of estimating equations such that the impact from estimating the high-dimensional nuisance parameters becomes asymptotically negligible. The new construction enables us to estimate a valid confidence region by empirical likelihood ratio. For the second one, we propose a test …
A numerically efficient inverse method for parametric model uncertainty identification using maximum likelihood estimation is presented. The goal is to identify a probability model for a fixed number of model parameters based on a set of experiments. To perform maximum likelihood estimation, the output probability density function is required. Forward propagation of input uncertainty is established combining Polynomial Chaos and moment matching. High-order moments of the output distribution are estimated using the generalized Polynomial Chaos framework. Next, a maximum entropy parametric distr…
The parametric Welch $t$-test and the non-parametric Wilcoxon-Mann-Whitney test are the most commonly used two independent sample means tests. More recent testing approaches include the non-parametric, empirical likelihood and exponential empirical likelihood. However, the applicability of these non-parametric likelihood testing procedures is limited partially because of their tendency to inflate the type I error in small sized samples. In order to circumvent the type I error problem, we propose simple calibrations using the $t$ distribution and bootstrapping. The two non-parametric likelihood…
Recently Liu and Wang derived the likelihood ratio test (LRT) statistic and its asymptotic distribution for testing equality of two multinomial distributions vs. the alternative that the second distribution is larger in terms of increasing convex order (ICX). ICX is less restrictive than stochastic order and is a notion that has found applications in insurance and actuarial science. In this paper we propose a new test for ICX. The new test has several advantages over the LRT and over any test procedure that depends on asymptotic theory for implementation. The advantages include the following: …
In this work, we study non-parametric hypothesis testing problem with distribution function constraints. The empirical likelihood ratio test has been widely used in testing problems with moment (in)equality constraints. However, some detection problems cannot be described using moment (in)equalities. We propose a distribution function constraint along with an empirical likelihood ratio test. This detector is applicable to a wide variety of robust parametric/non-parametric detection problems. Since the distribution function constraints provide a more exact description of the null hypothesis, th…
Inference based on the penalized density ratio model is proposed and studied. The model under consideration is specified by assuming that the log--likelihood function of two unknown densities is of some parametric form. The model has been extended to cover multiple samples problems while its theoretical properties have been investigated using large sample theory. A main application of the density ratio model is testing whether two, or more, distributions are equal. We extend these results by arguing that the penalized maximum empirical likelihood estimator has less mean square error than that …
University of Newcastle（纽卡斯尔大学）；Uppsala University（乌普萨拉大学）
This paper considers parameter estimation for nonlinear state-space models, which is an important but challenging problem. We address this challenge by employing a variational inference (VI) approach, which is a principled method that has deep connections to maximum likelihood estimation. This VI approach ultimately provides estimates of the model as solutions to an optimisation problem, which is deterministic, tractable and can be solved using standard optimisation tools. A specialisation of this approach for systems with additive Gaussian noise is also detailed. The proposed method is examin…
We consider here the identification of change-points on large-scale data streams. The objective is to find the most efficient way of combining information across data stream so that detection is possible under the smallest detectable change magnitude. The challenge comes from the sparsity of change-points when only a small fraction of data streams undergo change at any point in time. The most successful approach to the sparsity issue so far has been the application of hard thresholding such that only local scores from data streams exhibiting significant changes are considered and added. Howeve…
Global Parameter Identification and Uncertainty Quantification of reduced-order ADM1 using Maximum Likelihood Estimation and Bayesian Inference
DBFZ Deutsches Biomasseforschungszentrum gGmbH（德国生物质研究中心）；Chemnitz University of Technology（开姆尼茨工业大学）；Technische Universität Berlin（柏林工业大学）；University of Cambridge（剑桥大学）；Leipzig University（莱比锡大学）；FH Münster University of Applied Sciences（明斯特应用科学大学）
This paper considers the maximum generalized empirical likelihood (GEL) estimation and inference on parameters identified by high dimensional moment restrictions with weakly dependent data when the dimensions of the moment restrictions and the parameters diverge along with the sample size. The consistency with rates and the asymptotic normality of the GEL estimator are obtained by properly restricting the growth rates of the dimensions of the parameters and the moment restrictions, as well as the degree of data dependence. It is shown that even in the high dimensional time series setting, the …
We propose a differential geometric construction for families of low-rank covariance matrices, via interpolation on low-rank matrix manifolds. In contrast with standard parametric covariance classes, these families offer significant flexibility for problem-specific tailoring via the choice of "anchor" matrices for the interpolation. Moreover, their low-rank facilitates computational tractability in high dimensions and with limited data. We employ these covariance families for both interpolation and identification, where the latter problem comprises selecting the most representative member of t…
This paper considers the maximum likelihood estimation of factor models of high dimension, where the number of variables (N) is comparable with or even greater than the number of observations (T). An inferential theory is developed. We establish not only consistency but also the rate of convergence and the limiting distributions. Five different sets of identification conditions are considered. We show that the distributions of the MLE estimators depend on the identification restrictions. Unlike the principal components approach, the maximum likelihood estimator explicitly allows heteroskedasti…
A general dynamical cluster identification framework including both modeling and computation is developed. The earthquake declustering problem is studied to demonstrate how this framework applies. A stochastic model is proposed for earthquake occurrences that considers the sequence of occurrences as composed of two parts: earthquake clusters and single earthquakes. We suggest that earthquake clusters contain a ``mother quake'' and her ``offspring.'' Applying the filtering techniques, we use the solution of filtering equations as criteria for declustering. A procedure for calculating maximum li…
We consider the system identification problem of estimating a dynamical parameter of a Markovian quantum open system (the atom maser), by performing continuous time measurements in the system's output (outgoing atoms). Two estimation methods are investigated and compared. On the one hand, the maximum likelihood estimator (MLE) takes into account the full measurement data and is asymptotically optimal in terms of its mean square error. On the other hand, the `likelihood-free' method of approximate Bayesian computation (ABC) produces an approximation of the posterior distribution for a given set…
In this paper, we use convolutional neural networks to address the problem of model identification for autoregressive moving average time series models. We compare the performance of several neural network architectures, trained on simulated time series, with likelihood based methods, in particular the Akaike and Bayesian information criteria. We find that our neural networks can significantly outperform these likelihood based methods in terms of accuracy and, by orders of magnitude, in terms of speed.
This paper proposes a penalized composite likelihood method for model selection in colored graphical Gaussian models. The method provides a sparse and symmetry-constrained estimator of the precision matrix, and thus conducts model selection and precision matrix estimation simultaneously. In particular, the method uses penalty terms to constrain the elements of the precision matrix, which enables us to transform the model selection problem into a constrained optimization problem. Further, computer experiments are conducted to illustrate the performance of the proposed new methodology. It is sho…
Lennard (2013) [Fingerprint identification: how far have we come? Aus J Forensic Sci. doi:10.1080/00450618.2012.752037] proposes that the numeric output of statistical models should not be presented in court (except "if necessary" / "if required"). Instead he argues in favour of an "expert opinion" which may be informed by a statistical model but which is not itself the output of a statistical model. We argue that his proposed procedure lacks the transparency, the ease of testing of validity and reliability, and the relative robustness to cognitive bias that are the strengths of a likelihood-r…
When a missing-data mechanism is NMAR or non-ignorable, missingness is itself vital information and it must be taken into the likelihood, which, however, needs to introduce additional parameters to be estimated. The incompleteness of the data and introduction of more parameters can cause the identification problem. When a response variable is binary, it becomes a more serious problem because of less information of bi- nary data, however, there are no methods to briefly verify whether a mode is identified or not. Therefore, we provide a new necessary and sufficient condition to easily check mod…
We propose a new semiparametric approach for modelling nonlinear univariate diffusions, where the observed process is a nonparametric transformation of an underlying parametric diffusion (UPD). This modelling strategy yields a general class of semiparametric Markov diffusion models with parametric dynamic copulas and nonparametric marginal distributions. We provide primitive conditions for the identification of the UPD parameters together with the unknown transformations from discrete samples. Likelihood-based estimators of both parametric and nonparametric components are developed and we anal…
