Several strategies have been developed recently to ensure valid inference after model selection; some of these are easy to compute, while others fare better in terms of inferential power. In this paper, we consider a selective inference framework for Gaussian data. We propose a new method for inference through approximate maximum likelihood estimation. Our goal is to: (i) achieve better inferential power with the aid of randomization, (ii) bypass expensive MCMC sampling from exact conditional distributions that are hard to evaluate in closed forms. We construct approximate inference, e.g., p-values, confidence intervals etc., by solving a fairly simple, convex optimization problem. We illustrate the potential of our method across wide-ranging values of signal-to-noise ratio in simulations. On a cancer gene expression data set we find that our method improves upon the inferential power of some commonly used strategies for selective inference.
Statistical inference after model selection requires an inference framework that takes the selection into account in order to be valid. Following recent work on selective inference, we derive analytical expressions for inference after likelihood- or test-based model selection for linear models.
In multi-center clinical trials, due to various reasons, the individual-level data are strictly restricted to be assessed publicly. Instead, the summarized information is widely available from published results. With the advance of computational technology, it has become very common in data analyses to run on hundreds or thousands of machines simultaneous, with the data distributed across those machines and no longer available in a single central location. How to effectively assemble the summarized clinical data information or information from each machine in parallel computation has become a …
Maximum likelihood estimators are proposed for the parameters and the densities in a semiparametric density ratio model in which the nonparametric baseline density is approximated by the Bernstein polynomial model. The EM algorithm is used to obtain the maximum approximate Bernstein likelihood estimates. Simulation study shows that the performance of the proposed method is much better than the existing ones. The proposed method is illustrated by real data examples. Some asymptotic results are also presented and proved.
We are living in the big data era, as current technologies and networks allow for the easy and routine collection of data sets in different disciplines. Bayesian Statistics offers a flexible modeling approach which is attractive for describing the complexity of these datasets. These models often exhibit a likelihood function which is intractable due to the large sample size, high number of parameters, or functional complexity. Approximate Bayesian Computational (ABC) methods provides likelihood-free methods for performing statistical inferences with Bayesian models defined by intractable likel…
Know When to Abstain: Optimal Selective Classification with Likelihood Ratios
Department of Computer Science, National University of Singapore（计算机科学系，国立新加坡大学）；Smart Systems Institute, National University of Singapore（智能系统研究所，国立新加坡大学）
Tukey's $g$-and-$h$ distribution has been a powerful tool for data exploration and modeling since its introduction. However, two long standing challenges associated with this distribution family have remained unsolved until this day: how to find an optimal estimation procedure and how to make valid statistical inference on unknown parameters. To overcome these two challenges, a computationally efficient estimation procedure based on maximizing an approximated likelihood function of the Tukey's $g$-and-$h$ distribution is proposed and is shown to have the same estimation efficiency as the maxim…
Generalized linear mixed models (GLMMs) are used to model responses from exponential families with a combination of fixed and random effects. For variance components in GLMMs, we propose an approximate restricted likelihood ratio test that conducts testing on the working responses used in penalized quasi-likelihood estimation. This presents the hypothesis test in terms of normalized responses, allowing for application of existing testing methods for linear mixed models. Our test is flexible, computationally efficient, and outperforms several competitors. We illustrate the utility of the propos…
Continuous-time state-space models (SSMs) are flexible tools for analysing irregularly sampled sequential observations that are driven by an underlying state process. Corresponding applications typically involve restrictive assumptions concerning linearity and Gaussianity to facilitate inference on the model parameters via the Kalman filter. In this contribution, we provide a general continuous-time SSM framework, allowing both the observation and the state process to be non-linear and non-Gaussian. Statistical inference is carried out by maximum approximate likelihood estimation, where multip…
The ratio between two probability density functions is an important component of various tasks, including selection bias correction, novelty detection and classification. Recently, several estimators of this ratio have been proposed. Most of these methods fail if the sample space is high-dimensional, and hence require a dimension reduction step, the result of which can be a significant loss of information. Here we propose a simple-to-implement, fully nonparametric density ratio estimator that expands the ratio in terms of the eigenfunctions of a kernel-based operator; these functions reflect t…
As datasets grow larger, they are often distributed across multiple machines that compute in parallel and communicate with a central machine through short messages. In this paper, we focus on sparse regression and propose a new procedure for conducting selective inference with distributed data. Although many distributed procedures exist for point estimation in the sparse setting, few options are available for estimating uncertainties or conducting hypothesis tests based on the estimated sparsity. We solve a generalized linear regression on each machine, which then communicates a selected set o…
A maximum likelihood methodology for a general class of models is presented, using an approximate Bayesian computation (ABC) approach. The typical target of ABC methods are models with intractable likelihoods, and we combine an ABC-MCMC sampler with so-called "data cloning" for maximum likelihood estimation. Accuracy of ABC methods relies on the use of a small threshold value for comparing simulations from the model and observed data. The proposed methodology shows how to use large threshold values, while the number of data-clones is increased to ease convergence towards an approximate maximum…
In this paper, we mainly focus on the penalized maximum likelihood estimation (MLE) of the high-dimensional approximate factor model. Since the current estimation procedure can not guarantee the positive definiteness of the error covariance matrix, by reformulating the estimation of error covariance matrix and based on the lagrangian duality, we propose an accelerated proximal gradient (APG) algorithm to give a positive definite estimate of the error covariance matrix. Combined the APG algorithm with EM method, a new estimation procedure is proposed to estimate the high-dimensional approximate…
Complex phenomena in engineering and the sciences are often modeled with computationally intensive feed-forward simulations for which a tractable analytic likelihood does not exist. In these cases, it is sometimes necessary to estimate an approximate likelihood or fit a fast emulator model for efficient statistical inference; such surrogate models include Gaussian synthetic likelihoods and more recently neural density estimators such as autoregressive models and normalizing flows. To date, however, there is no consistent way of quantifying the quality of such a fit. Here we propose a statistic…
Empirical economic research frequently applies maximum likelihood estimation in cases where the likelihood function is analytically intractable. Most of the theoretical literature focuses on maximum simulated likelihood (MSL) estimators, while empirical and simulation analyzes often find that alternative approximation methods such as quasi-Monte Carlo simulation, Gaussian quadrature, and integration on sparse grids behave considerably better numerically. This paper generalizes the theoretical results widely known for MSL estimators to a general set of maximum approximated likelihood (MAL) esti…
Temporal Point Processes (TPP) with partial likelihoods involving a latent structure often entail an intractable marginalization, thus making inference hard. We propose a novel approach to Maximum Likelihood Estimation (MLE) involving approximate inference over the latent variables by minimizing a tight upper bound on the approximation gap. Given a discrete latent variable $Z$, the proposed approximation reduces inference complexity from $O(|Z|^c)$ to $O(|Z|)$. We use convex conjugates to determine this upper bound in a closed form and show that its addition to the optimization objective resul…
In this paper we consider regression problems subject to arbitrary noise in the operator or design matrix. This characterization appropriately models many physical phenomena with uncertainty in the regressors. Although the problem has been studied extensively for ordinary/total least squares, and via models that implicitly or explicitly assume Gaussianity, less attention has been paid to improving estimation for regression problems under general uncertainty in the design matrix. To address difficulties encountered when dealing with distributions of sums of random variables, we rely on the sadd…
A trigonometrically approximated maximum likelihood estimation for $α$-stable laws is proposed. The estimator solves the approximated likelihood equation, which is obtained by projecting a true score function on the space spanned by trigonometric functions. The projected score is expressed only by real and imaginary parts of the characteristic function and their derivatives, so that we can explicitly construct the targeting estimating equation. We study the asymptotic properties of the proposed estimator and show consistency and asymptotic normality. Furthermore, as the number of trigonometric…
The statistical analysis of discrete data has been the subject of extensive statistical research dating back to the work of Pearson. In this survey we review some recently developed methods for testing hypotheses about high-dimensional multinomials. Traditional tests like the $χ^2$ test and the likelihood ratio test can have poor power in the high-dimensional setting. Much of the research in this area has focused on finding tests with asymptotically Normal limits and developing (stringent) conditions under which tests have Normal limits. We argue that this perspective suffers from a significan…
Parametric inference posits a statistical model that is a specified family of probability distributions. Restricted inference, e.g., restricted likelihood ratio testing, attempts to exploit the structure of a statistical submodel that is a subset of the specified family. We consider the problem of testing a simple hypothesis against alternatives from such a submodel. In the case of an unknown submodel, it is not clear how to realize the benefits of restricted inference. To do so, we first construct information tests that are locally asymptotically equivalent to likelihood ratio tests. Informat…
University of Warsaw（华沙大学）；Polish National Institute for Machine Learning（波兰国家机器学习研究所）；Applica；Jagiellonian University（雅盖隆大学）；University of Oxford（牛津大学）
Most of the existing methods for estimating the local intrinsic dimension of a data distribution do not scale well to high-dimensional data. Many of them rely on a non-parametric nearest neighbors approach which suffers from the curse of dimensionality. We attempt to address that challenge by proposing a novel approach to the problem: Local Intrinsic Dimension estimation using approximate Likelihood (LIDL). Our method relies on an arbitrary density estimation method as its subroutine and hence tries to sidestep the dimensionality challenge by making use of the recent progress in parametric neu…
The transition density of a diffusion process does not admit an explicit expression in general, which prevents the full maximum likelihood estimation (MLE) based on discretely observed sample paths. A\"ıt-Sahalia [J. Finance 54 (1999) 1361--1395; Econometrica 70 (2002) 223--262] proposed asymptotic expansions to the transition densities of diffusion processes, which lead to an approximate maximum likelihood estimation (AMLE) for parameters. Built on A\"ıt-Sahalia's [Econometrica 70 (2002) 223--262; Ann. Statist. 36 (2008) 906--937] proposal and analysis on the AMLE, we establish the consistenc…
Linear ARCH (LARCH) processes were introduced by Robinson [J. Econometrics 47 (1991) 67--84] to model long-range dependence in volatility and leverage. Basic theoretical properties of LARCH processes have been investigated in the recent literature. However, there is a lack of estimation methods and corresponding asymptotic theory. In this paper, we consider estimation of the dependence parameters for LARCH processes with non-summable hyperbolically decaying coefficients. Asymptotic limit theorems are derived. A central limit theorem with $\sqrt{n}$-rate of convergence holds for an approximate …
An approximate maximum likelihood method of estimation of diffusion parameters $(\vartheta,σ)$ based on discrete observations of a diffusion $X$ along fixed time-interval $[0,T]$ and Euler approximation of integrals is analyzed. We assume that $X$ satisfies a SDE of form $dX_t =μ(X_t ,\vartheta )\, dt+\sqrtσ b(X_t )\, dW_t$, with non-random initial condition. SDE is nonlinear in $\vartheta$ generally. Based on assumption that maximum likelihood estimator $\hat{\vartheta}_T$ of the drift parameter based on continuous observation of a path over $[0,T]$ exists we prove that measurable estimator $…
In recent years, methods of approximate parameter estimation have attracted considerable interest in complex problems where exact likelihoods are hard to obtain. In their most basic form, Bayesian methods such as Approximate Bayesian Computation (ABC) involve sampling from the parameter space and keeping those parameters that produce data that fit sufficiently well to the actually observed data. Exploring the whole parameter space, however, makes this approach inefficient in high dimensional problems. This led to the proposal of more sophisticated iterative methods of inference such as particl…
Quantile regression is often used when a comprehensive relationship between a response variable and one or more explanatory variables is desired. The traditional frequentists' approach to quantile regression has been well developed around asymptotic theories and efficient algorithms. However, not much work has been published under the Bayesian framework. One challenging problem for Bayesian quantile regression is that the full likelihood has no parametric forms. In this paper, we propose a Bayesian quantile regression method, the linearly interpolated density (LID) method, which uses a linear …
Different change-point type models encountered in statistical inference for stochastic processes give rise to different limiting likelihood ratio processes. In this paper we consider two such likelihood ratios. The first one is an exponential functional of a two-sided Poisson process driven by some parameter, while the second one is an exponential functional of a two-sided Brownian motion. We establish that for sufficiently small values of the parameter, the Poisson type likelihood ratio can be approximated by the Brownian type one. As a consequence, several statistically interesting quantitie…
University of California, San Francisco（加利福尼亚大学旧金山分校）；Flatiron Health；University of Washington（华盛顿大学）
Though black-box predictors are state-of-the-art for many complex tasks, they often fail to properly quantify predictive uncertainty and may provide inappropriate predictions for unfamiliar data. Instead, we can learn more reliable models by letting them either output a prediction set or abstain when the uncertainty is high. We propose training these selective prediction-set models using an uncertainty-aware loss minimization framework, which unifies ideas from decision theory and robust maximum likelihood. Moreover, since black-box methods are not guaranteed to output well-calibrated predicti…
In many fields of science, generalized likelihood ratio tests are established tools for statistical inference. At the same time, it has become increasingly common that a simulator (or generative model) is used to describe complex processes that tie parameters $θ$ of an underlying theory and measurement apparatus to high-dimensional observations $\mathbf{x}\in \mathbb{R}^p$. However, simulator often do not provide a way to evaluate the likelihood function for a given observation $\mathbf{x}$, which motivates a new class of likelihood-free inference algorithms. In this paper, we show that likeli…
If the log likelihood is approximately quadratic with constant Hessian, then the maximum likelihood estimator (MLE) is approximately normally distributed. No other assumptions are required. We do not need independent and identically distributed data. We do not need the law of large numbers (LLN) or the central limit theorem (CLT). We do not need sample size going to infinity or anything going to infinity. Presented here is a combination of Le Cam style theory involving local asymptotic normality (LAN) and local asymptotic mixed normality (LAMN) and Cramér style theory involving derivatives and…
Different change-point type models encountered in statistical inference for stochastic processes give rise to different limiting likelihood ratio processes. In a previous paper of one of the authors it was established that one of these likelihood ratios, which is an exponential functional of a two-sided Poisson process driven by some parameter, can be approximated (for sufficiently small values of the parameter) by another one, which is an exponential functional of a two-sided Brownian motion. In this paper we consider yet another likelihood ratio, which is the exponent of a two-sided compound…
Maximum approximate Bernstein likelihood estimates of the baseline density function and the regression coefficients in the proportional hazard regression models based on interval-censored event time data are proposed. This results in not only a smooth estimate of the survival function which enjoys faster convergence rate but also improved estimates of the regression coefficients. Simulation shows that the finite sample performance of the proposed method is better than the existing ones. The proposed method is illustrated by real data applications.
Latent variable models represent a useful tool for the analysis of complex data when the constructs of interest are not observable. A problem related to these models is that the integrals involved in the likelihood function cannot be solved analytically. We propose a computational approach, referred to as Dimension Reduction Method (DRM), that consists of a dimension reduction of the multidimensional integral that makes the computation feasible in situations in which the quadrature based methods are not applicable. We discuss the advantages of DRM compared with other existing approximation pro…
Synthetic likelihood is an attractive approach to likelihood-free inference when an approximately Gaussian summary statistic for the data, informative for inference about the parameters, is available. The synthetic likelihood method derives an approximate likelihood function from a plug-in normal density estimate for the summary statistic, with plug-in mean and covariance matrix obtained by Monte Carlo simulation from the model. In this article, we develop alternatives to Markov chain Monte Carlo implementations of Bayesian synthetic likelihoods with reduced computational overheads. Our approa…
We study the estimation of a high dimensional approximate factor model in the presence of both cross sectional dependence and heteroskedasticity. The classical method of principal components analysis (PCA) does not efficiently estimate the factor loadings or common factors because it essentially treats the idiosyncratic error to be homoskedastic and cross sectionally uncorrelated. For efficient estimation it is essential to estimate a large error covariance matrix. We assume the model to be conditionally sparse, and propose two approaches to estimating the common factors and factor loadings; b…
Nonparametric maximum likelihood (NPML) for mixture models is a technique for estimating mixing distributions that has a long and rich history in statistics going back to the 1950s, and is closely related to empirical Bayes methods. Historically, NPML-based methods have been considered to be relatively impractical because of computational and theoretical obstacles. However, recent work focusing on approximate NPML methods suggests that these methods may have great promise for a variety of modern applications. Building on this recent work, a class of flexible, scalable, and easy to implement ap…
Approximate Bayesian Computation (ABC) can be viewed as an analytic approximation of an intractable likelihood coupled with an elementary simulation step. Such a view, combined with a suitable instrumental prior distribution permits maximum-likelihood (or maximum-a-posteriori) inference to be conducted, approximately, using essentially the same techniques. An elementary approach to this problem which simply obtains a nonparametric approximation of the likelihood surface which is then used as a smooth proxy for the likelihood in a subsequent maximisation step is developed here and the convergen…
A Bayesian design is given by maximising an expected utility over a design space. The utility is chosen to represent the aim of the experiment and its expectation is taken with respect to all unknowns: responses, parameters and/or models. Although straightforward in principle, there are several challenges to finding Bayesian designs in practice. Firstly, the utility and expected utility are rarely available in closed form and require approximation. Secondly, the design space can be of high-dimensionality. In the case of intractable likelihood models, these problems are compounded by the fact t…
