We study the distribution of the maximum likelihood estimate (MLE) in high-dimensional logistic models, extending the recent results from Sur (2019) to the case where the Gaussian covariates may have an arbitrary covariance structure. We prove that in the limit of large problems holding the ratio between the number $p$ of covariates and the sample size $n$ constant, every finite list of MLE coordinates follows a multivariate normal distribution. Concretely, the $j$th coordinate $\hat β_j$ of the MLE is asymptotically normally distributed with mean $α_\star β_j$ and standard deviation $σ_\star/τ_j$; here, $β_j$ is the value of the true regression coefficient, and $τ_j$ the standard deviation of the $j$th predictor conditional on all the others. The numerical parameters $α_\star > 1$ and $σ_\star$ only depend upon the problem dimensionality $p/n$ and the overall signal strength, and can be accurately estimated. Our results imply that the MLE's magnitude is biased upwards and that the MLE's standard deviation is greater than that predicted by classical theory. We present a series of experiments on simulated and real data showing excellent agreement with the theory.
This short note is to point the reader to notice that the proof of high dimensional asymptotic normality of MLE estimator for logistic regression under the regime $p_n=o(n)$ given in paper: "Maximum likelihood estimation in logistic regression models with a diverging number of covariates. Electronic Journal of Statistics, 6, 1838-1846." is wrong.
This paper rigorously establishes that the existence of the maximum likelihood estimate (MLE) in high-dimensional logistic regression models with Gaussian covariates undergoes a sharp `phase transition'. We introduce an explicit boundary curve $h_{\text{MLE}}$, parameterized by two scalars measuring the overall magnitude of the unknown sequence of regression coefficients, with the following property: in the limit of large sample sizes $n$ and number of features $p$ proportioned in such a way that $p/n \rightarrow κ$, we show that if the problem is sufficiently high dimensional in the sense tha…
Every student in statistics or data science learns early on that when the sample size largely exceeds the number of variables, fitting a logistic model produces estimates that are approximately unbiased. Every student also learns that there are formulas to predict the variability of these estimates which are used for the purpose of statistical inference; for instance, to produce p-values for testing the significance of regression coefficients. Although these formulas come from large sample asymptotics, we are often told that we are on reasonably safe grounds when $n$ is large in such a way tha…
In matrix-valued datasets the sampled matrices often exhibit correlations among both their rows and their columns. A useful and parsimonious model of such dependence is the matrix normal model, in which the covariances among the elements of a random matrix are parameterized in terms of the Kronecker product of two covariance matrices, one representing row covariances and one representing column covariance. An appealing feature of such a matrix normal model is that the Kronecker covariance structure allows for standard likelihood inference even when only a very small number of data matrices is …
Finite-sample performance of the maximum likelihood estimator in logistic regression
Department of Statistics, University of California, Berkeley, USA（加州大学伯克利分校统计学系）；Department of Statistics, CREST/ENSAE Paris, Palaiseau, France（巴黎中央银行与应用经济学院统计学系）
In cluster-specific studies, ordinary logistic regression and conditional logistic regression for binary outcomes provide maximum likelihood estimator (MLE) and conditional maximum likelihood estimator (CMLE), respectively. In this paper, we show that CMLE is approaching to MLE asymptotically when each individual data point is replicated infinitely many times. Our theoretical derivation is based on the observation that a term appearing in the conditional average log-likelihood function is the coefficient of a polynomial, and hence can be transformed to a complex integral by Cauchy's differenti…
Estimating the unconstrained mean and covariance matrix is a popular topic in statistics. However, estimation of the parameters of $N_p(μ,Σ)$ under joint constraints such as $Σμ= μ$ has not received much attention. It can be viewed as a multivariate counterpart of the classical estimation problem in the $N(θ,θ^2)$ distribution. In addition to the usual inference challenges under such non-linear constraints among the parameters (curved exponential family), one has to deal with the basic requirements of symmetry and positive definiteness when estimating a covariance matrix. We derive the non-lin…
When the spatial sample size is extremely large, which occurs in many environmental and ecological studies, operations on the large covariance matrix are a numerical challenge. Covariance tapering is a technique to alleviate the numerical challenges. Under the assumption that data are collected along a line in a bounded region, we investigate how the tapering affects the asymptotic efficiency of the maximum likelihood estimator (MLE) for the microergodic parameter in the Matérn covariance function by establishing the fixed-domain asymptotic distribution of the exact MLE and that of the tapered…
Inference about dependencies in a multiway data array can be made using the array normal model, which corresponds to the class of multivariate normal distributions with separable covariance matrices. Maximum likelihood and Bayesian methods for inference in the array normal model have appeared in the literature, but there have not been any results concerning the optimality properties of such estimators. In this article, we obtain results for the array normal model that are analogous to some classical results concerning covariance estimation for the multivariate normal model. We show that under …
We consider goodness-of-fit tests of symmetric stable distributions based on weighted integrals of the squared distance between the empirical characteristic function of the standardized data and the characteristic function of the standard symmetric stable distribution with the characteristic exponent $α$ estimated from the data. We treat $α$ as an unknown parameter, but for theoretical simplicity we also consider the case that $α$ is fixed. For estimation of parameters and the standardization of data we use maximum likelihood estimator (MLE) and an equivariant integrated squared error estimato…
Consider a set of categorical variables $\mathcal{P}$ where at least one, denoted by $Y$, is binary. The log-linear model that describes the counts in the resulting contingency table implies a specific logistic regression model, with the binary variable as the outcome. Extending results in Christensen (1997), by also considering the case where factors present in the contingency table disappear from the logistic regression model, we prove that the Maximum Likelihood Estimate (MLE) for the parameters of the logistic regression equals the MLE for the corresponding parameters of the log-linear mod…
Sample covariance matrices are widely used in multivariate statistical analysis. The central limit theorems (CLT's) for linear spectral statistics of high-dimensional non-centered sample covariance matrices have received considerable attention in random matrix theory and have been applied to many high-dimensional statistical problems. However, known population mean vectors are assumed for non-centered sample covariance matrices, some of which even assume Gaussian-like moment conditions. In fact, there are still another two most frequently used sample covariance matrices: the MLE (by subtractin…
This article presents maximum likelihood estimators (MLEs) and log-likelihood ratio (LLR) tests for the eigenvalues and eigenvectors of Gaussian random symmetric matrices of arbitrary dimension, where the observations are independent repeated samples from one or two populations. These inference problems are relevant in the analysis of diffusion tensor imaging data and polarized cosmic background radiation data, where the observations are, respectively, $3\times3$ and $2\times2$ symmetric positive definite matrices. The parameter sets involved in the inference problems for eigenvalues and eigen…
This paper studies binary logistic regression for rare events data, or imbalanced data, where the number of events (observations in one class, often called cases) is significantly smaller than the number of nonevents (observations in the other class, often called controls). We first derive the asymptotic distribution of the maximum likelihood estimator (MLE) of the unknown parameter, which shows that the asymptotic variance convergences to zero in a rate of the inverse of the number of the events instead of the inverse of the full data sample size. This indicates that the available information…
Logistic regression remains one of the most widely used tools in applied statistics, machine learning and data science. However, in moderately high-dimensional problems, where the number of features $d$ is a non-negligible fraction of the sample size $n$, the logistic regression maximum likelihood estimator (MLE), and statistical procedures based the large-sample approximation of its distribution, behave poorly. Recently, Sur and Candès (2019) showed that these issues can be corrected by applying a new approximation of the MLE's sampling distribution in this high-dimensional regime. Unfortunat…
Markov regime switching models have been widely used in numerous empirical applications in economics and finance. However, the asymptotic distribution of the maximum likelihood estimator (MLE) has not been proven for some empirically popular Markov regime switching models. In particular, the asymptotic distribution of the MLE has been unknown for models in which some elements of the transition probability matrix have the value of zero, as is commonly assumed in empirical applications with models with more than two regimes. This also includes models in which the regime-specific density depends …
The L1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix, or alternatively the underlying graph structure of a Gaussian Markov Random Field, from very limited samples. We propose a novel algorithm for solving the resulting optimization problem which is a regularized log-determinant program. In contrast to recent state-of-the-art methods that largely use first order gradient information, our algorithm is based on Newton's method and employs a quadratic approximation, but with some modific…
Accurate statistical inference in logistic regression models remains a critical challenge when the ratio between the number of parameters and sample size is not negligible. This is because approximations based on either classical asymptotic theory or bootstrap calculations are grossly off the mark. This paper introduces a resized bootstrap method to infer model parameters in arbitrary dimensions. As in the parametric bootstrap, we resample observations from a distribution, which depends on an estimated regression coefficient sequence. The novelty is that this estimate is actually far from the …
We analyze the problem of maximum likelihood estimation for Gaussian distributions that are multivariate totally positive of order two (MTP2). By exploiting connections to phylogenetics and single-linkage clustering, we give a simple proof that the maximum likelihood estimator (MLE) for such distributions exists based on at least 2 observations, irrespective of the underlying dimension. Slawski and Hein, who first proved this result, also provided empirical evidence showing that the MTP2 constraint serves as an implicit regularizer and leads to sparsity in the estimated inverse covariance matr…
This paper studies the problem of statistical inference for genetic relatedness between binary traits based on individual-level genome-wide association data. Specifically, under the high-dimensional logistic regression models, we define parameters characterizing the cross-trait genetic correlation, the genetic covariance and the trait-specific genetic variance. A novel weighted debiasing method is developed for the logistic Lasso estimator and computationally efficient debiased estimators are proposed. The rates of convergence for these estimators are studied and their asymptotic normality is …
This paper is concerned with the problem of top-$K$ ranking from pairwise comparisons. Given a collection of $n$ items and a few pairwise comparisons across them, one wishes to identify the set of $K$ items that receive the highest ranks. To tackle this problem, we adopt the logistic parametric model --- the Bradley-Terry-Luce model, where each item is assigned a latent preference score, and where the outcome of each pairwise comparison depends solely on the relative scores of the two items involved. Recent works have made significant progress towards characterizing the performance (e.g. the m…
Covariance alignment: from maximum likelihood estimation to Gromov-Wasserstein
Center for Data Science, New York University（纽约大学数据科学中心）；Institute for Data, Systems, and Society, Massachusetts Institute of Technology（麻省理工学院数据、系统与社会研究所）；Department of Mathematics, Massachusetts Institute of Technology（麻省理工学院数学系）
In this paper, we present a novel framework incorporating a combination of sparse models in different domains. We posit the observed data as generated from a linear combination of a sparse Gaussian Markov model (with a sparse precision matrix) and a sparse Gaussian independence model (with a sparse covariance matrix). We provide efficient methods for decomposition of the data into two domains, \viz Markov and independence domains. We characterize a set of sufficient conditions for identifiability and model consistency. Our decomposition method is based on a simple modification of the popular $…
Fitting high-dimensional data involves a delicate tradeoff between faithful representation and the use of sparse models. Too often, sparsity assumptions on the fitted model are too restrictive to provide a faithful representation of the observed data. In this paper, we present a novel framework incorporating sparsity in different domains.We decompose the observed covariance matrix into a sparse Gaussian Markov model (with a sparse precision matrix) and a sparse independence model (with a sparse covariance matrix). Our framework incorporates sparse covariance and sparse precision estimation as …
In this paper, the quantile based flattened logistic distribution introduced by Gilchrist has been studied. Some classical and quantile based properties of the distribution have been obtained. Closed form expression of L-moments and L-ratios of the distribution have been obtained. A quantile based analysis based on the methods of matching L-moments estimation is employed to estimate the parameters of the proposed model. We further derive the asymptotic variance-covariance matrix of the L-Moments estimator of the proposed model. Finally, we apply the proposed model to a real data set and perfor…
Maximum likelihood estimation is a popular method in statistical inference. As a way of assessing the accuracy of the maximum likelihood estimate (MLE), the calculation of the covariance matrix of the MLE is of great interest in practice. Standard statistical theory shows that the normalized MLE is asymptotically normally distributed with covariance matrix being the inverse of the Fisher information matrix (FIM) at the unknown parameter. Two commonly used estimates for the covariance of the MLE are the inverse of the observed FIM (the same as the inverse Hessian of the negative log-likelihood)…
In this article, we introduce a two-way factor model for a high-dimensional data matrix and study the properties of the maximum likelihood estimation (MLE). The proposed model assumes separable effects of row and column attributes and captures the correlation across rows and columns with low-dimensional hidden factors. The model inherits the dimension-reduction feature of classical factor models but introduces a new framework with separable row and column factors, representing the covariance or correlation structure in the data matrix. We propose a block alternating, maximizing strategy to com…
The association between two random variables is often of primary interest in statistical research. In this paper semiparametric models for the association between random vectors X and Y are considered which leave the marginal distributions arbitrary. Given that the odds ratio function comprises the whole information about the association the focus is on bilinear log-odds ratio models and in particular on the odds ratio parameter vector θ. The covariance structure of the maximum likelihood estimator θ^ of θ is of major importance for asymptotic inference. To this end different representations o…
For many diseases, logistic and other constraints often render large incidence studies difficult, if not impossible, to carry out. This becomes a drawback, particularly when a new incidence study is needed each time the disease incidence rate is investigated in a different population. However, by carrying out a prevalent cohort study with follow-up it is possible to estimate the incidence rate if it is constant. In this paper we derive the maximum likelihood estimator (MLE) of the overall incidence rate, $λ$, as well as age-specific incidence rates, by exploiting the well known epidemiologic r…
This paper develops a new class of conditional Markov jump processes with regime switching and paths dependence. The key novel feature of the developed process lies on its ability to switch the transition rate as it moves from one state to another with switching probability depending on the current state and time of the process as well as its past trajectories. As such, the transition from current state to another depends on the holding time of the process in the state. Distributional properties of the process are given explicitly in terms of the speed regimes represented by a finite number of…
In this paper, we mainly focus on the penalized maximum likelihood estimation (MLE) of the high-dimensional approximate factor model. Since the current estimation procedure can not guarantee the positive definiteness of the error covariance matrix, by reformulating the estimation of error covariance matrix and based on the lagrangian duality, we propose an accelerated proximal gradient (APG) algorithm to give a positive definite estimate of the error covariance matrix. Combined the APG algorithm with EM method, a new estimation procedure is proposed to estimate the high-dimensional approximate…
We explore the performance of sample average approximation in comparison with several other methods for stochastic optimization when there is information available on the underlying true probability distribution. The methods we evaluate are (a) bagging; (b) kernel smoothing; (c) maximum likelihood estimation (MLE); and (d) a Bayesian approach. We use two test sets, the first has a quadratic objective function allowing for very different types of interaction between the random component and the univariate decision variable. Here the sample average approximation is remarkably effective and only …
Near optimal sample complexity for matrix and tensor normal models via geodesic convexity
Department of Mathematics, Massachusetts Institute of Technology（麻省理工学院数学系）；Cheriton School of Computer Science, University of Waterloo（滑铁卢大学丘成桐计算机科学学院）；Computer Science Department, University of British Columbia（不列颠哥伦比亚大学计算机科学系）；Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学）
This paper considers the problem of robustly estimating the parameters of a heavy-tailed multivariate distribution when the covariance matrix is known to have the structure of a low-rank matrix plus a diagonal matrix as considered in factor analysis (FA). By assuming the observed data to follow the multivariate Student's t distribution, we can robustly estimate the parameters via maximum likelihood estimation (MLE). However, the MLE of parameters becomes an intractable problem when the multivariate Student's t distribution and the FA structure are both introduced. In this paper, we propose an …
In the present paper, progression of non-alcoholic fatty liver disease (NAFLD) process is modeled by Continuous time Markov chains (CTMC) with 4 states .The transition intensities among the states are estimated using maximum likelihood estimation (MLE) method. The transition probabilities are also calculated. The mean sojourn time and its variance are estimated as well as the state probability distribution and its asymptotic covariance matrix. The life expectancy of the patient, one of the important statistical indices, is also obtained. The paper illustrates the new approach of using MLE to c…
In this paper we consider the behavior of Kalman Filter state estimates in the case of distribution with heavy tails .The simulated linear state space models with Gaussian measurement noises were used. Gaussian noises in state equation are replaced by components with alpha-stable distribution with di erent parameters alpha and beta. We consider the case when "all parameters are known" and two methods of parameters estimation are compared: the maximum likelihood estimator (MLE) and the expectation- maximization algorithm (EM). It was shown that in cases of large deviation from Gaussian distribu…
Graphical models are a framework for representing and exploiting prior conditional independence structures within distributions using graphs. In the Gaussian case, these models are directly related to the sparsity of the inverse covariance (concentration) matrix and allow for improved covariance estimation with lower computational complexity. We consider concentration estimation with the mean-squared error (MSE) as the objective, in a special type of model known as decomposable. This model includes, for example, the well known banded structure and other cases encountered in practice. Our first…
In regression models for spatial data, it is often assumed that the marginal effects of covariates on the response are constant over space. In practice, this assumption might often be questionable. In this article, we show how a Gaussian process-based spatially varying coefficient (SVC) model can be estimated using maximum likelihood estimation (MLE). In addition, we present an approach that scales to large data by applying covariance tapering. We compare our methodology to existing methods such as a Bayesian approach using the stochastic partial differential equation (SPDE) link, geographical…
Hierarchical autocorrelation in the error term of linear models arises when sampling units are related to each other according to a tree. The residual covariance is parametrized using the tree-distance between sampling units. When observations are modeled using an Ornstein-Uhlenbeck (OU) process along the tree, the autocorrelation between two tips decreases exponentially with their tree distance. These models are most often applied in evolutionary biology, when tips represent biological species and the OU process parameters represent the strength and direction of natural selection. For these m…
