Cosma Rohilla Shalizi
This expository note describes how to apply the method of maximum likelihood to estimate the parameters of the ``$q$-exponential'' distributions introduced by Tsallis and collaborators. It also describes the relationship of these distributions to the classical Pareto distributions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
Tsallis entropy is a generalized diversity index first derived in Patil and Taillie (1982) and then rediscovered in community ecology by Keylock (2005). Bayesian nonparametric estimation of Shannon entropy and Simpson's diversity under uniform and symmetric Dirichlet priors has been already advocated as an alternative to maximum likelihood estimation based on frequency counts, which is negatively biased in the undersampled regime. Here we present a fully general Bayesian nonparametric estimation of the whole class of Tsallis diversity indices under Gnedin-Pitman priors, a large family of rando…
Estimation of Tsallis entropy and its applications to goodness-of-fit tests
本文提出基于高阶样本间距和分位数函数的Tsallis熵估计方法，并应用于正态和指数分布的拟合优度检验，通过模拟验证其有效性。
Entropy and relative or cross entropy measures are two very fundamental concepts in information theory and are also widely used for statistical inference across disciplines. The related optimization problems, in particular the maximization of the entropy and the minimization of the cross entropy or relative entropy (divergence), are essential for general logical inference in our physical world. In this paper, we discuss a two parameter generalization of the popular Rényi entropy and associated optimization problems. We derive the desired entropic characteristics of the new generalized entropy …
In a regular full exponential family, the maximum likelihood estimator (MLE) need not exist in the traditional sense. However, the MLE may exist in the completion of the exponential family. Existing algorithms for finding the MLE in the completion solve many linear programs; they are slow in small problems and too slow for large problems. We provide new, fast, and scalable methodology for finding the MLE in the completion of the exponential family. This methodology is based on conventional maximum likelihood computations which come close, in a sense, to finding the MLE in the completion of the…
Maximum likelihood estimation for the $λ$-exponential family
本文针对$λ$-指数族提出一种基于不动点迭代的最大似然估计算法，并利用广义凸对偶证明似然单调性，通过$q$-高斯分布和Dirichlet扰动验证了方法。
On variational inference and maximum likelihood estimation with the λ-exponential family
针对λ-指数族研究不足但应用需求广泛的问题，本文提出求解其变分推断与最大似然估计的理论算法框架，给出新最优性条件与类近端算法，数值实验验证其在重尾分布场景的有效性。
We characterize the existence of the maximum likelihood estimator for discrete exponential families. Our criterion is simple to apply as we show in various settings, most notably for exponential models of random graphs. As an application, we point out the size of independent identically distributed samples for which the maximum likelihood estimator exists with high probability.
Here, in this paper it has been considered a sub family of exponential family. Maximum likelihood estimations (MLE) for the parameter of this family, probability density function, and cumulative density function based on a sample and based on lower record values have been obtained. It has been considered Mean Square Error (MSE) as a criterion for determining which is better in different situations. Additionally, it has been proved some theories about the relations between MLE based on lower record values and based on a random sample. Also, some interesting asymptotically properties for these e…
In this paper a useful subfamily of the exponential family has been considered. The ML estimation based on upper record values has been calculated for the parameter, Cumulative Density Function, and Probability Density Function of the family. Also, the relations between MLE based on record values and a random sample has been discussed. Additionally, some properties of these estimators have been investigated. Finally, it has been proven that these estimators have some useful properties for samples with large size
Deriving Lehmer and Hölder means as maximum weighted likelihood estimates for the multivariate exponential family
本文将 Lehmer 和 Hölder 均值族与加权最大似然估计的联系从一元指数族推广到多元指数族，为这些均值族提供概率解释并拓展其应用。
This paper investigates the asymptotic theory of the quasi-maximum exponential likelihood estimators (QMELE) for ARMA--GARCH models. Under only a fractional moment condition, the strong consistency and the asymptotic normality of the global self-weighted QMELE are obtained. Based on this self-weighted QMELE, the local QMELE is showed to be asymptotically normal for the ARMA model with GARCH (finite variance) and IGARCH errors. A formal comparison of two estimators is given for some cases. A simulation study is carried out to assess the performance of these estimators, and a real example on the…
Context-sensitive hypothesis-testing and exponential families
本文提出基于样本字符串值的“加权”统计推断概念，将其应用于假设检验与指数族分析，并推广了标准统计中的经典不等式与散度。
The normality assumption on data set is very restrictive approach for modelling. The generalized form of normal distribution, named as an exponential power (EP) distribution, and its scale mixture form have been considered extensively to overcome the problem for modelling non-normal data set since last decades. However, examining the robustness properties of maximum likelihood (ML) estimators of parameters in these distributions, such as the in uence function, gross-error sensitivity, breakdown point and information-standardized sensitivity, has not been considered together. The well-known asy…
When in a full exponential family the maximum likelihood estimate (MLE) does not exist, the MLE may exist in the Barndorff-Nielsen completion of the family. We propose a practical algorithm for finding the MLE in the completion based on repeated linear programming using the R contributed package rcdd and illustrate it with two generalized linear model examples. When the MLE for the null hypothesis lies in the completion, likelihood ratio tests of model comparison are almost unchanged from the usual case. Only the degrees of freedom need to be adjusted. When the MLE lies in the completion, conf…
In finite mixtures of location-scale distributions, if there is no constraint on the parameters then the maximum likelihood estimate does not exist. But when the ratios of the scale parameters are restricted appropriately, the maximum likelihood estimate exists. We prove that the maximum likelihood estimator (MLE) is strongly consistent, if the ratios of the scale parameters are restricted from below by $\exp(-n^{d}), 0 < d < 1 $, where $n$ is the sample size.
Generalized linear mixed models are powerful tools for analyzing clustered data, where the unknown parameters are classically (and most commonly) estimated by the maximum likelihood and restricted maximum likelihood procedures. However, since the likelihood based procedures are known to be highly sensitive to outliers, M-estimators have become popular as a means to obtain robust estimates under possible data contamination. In this paper, we prove that, for sufficiently smooth general loss functions defining the M-estimators in generalized linear mixed models, the tail probability of the deviat…
In a finite mixture of location-scale distributions maximum likelihood estimator does not exist because of the unboundedness of the likelihood function when the scale parameter of some mixture component approaches zero. In order to study the strong consistency of maximum likelihood estimator, we consider the case that the scale parameters of the component distributions are restricted from below by $c_n$, where $c_n$ is a sequence of positive real numbers which tend to zero as the sample size $n$ increases. We prove that under mild regularity conditions maximum likelihood estimator is strongly …
We consider maximum likelihood estimation of finite mixture of uniform distributions. We prove that maximum likelihood estimator is strongly consistent, if the scale parameters of the component uniform distributions are restricted from below by exp(-n^d), 0 < d < 1, where n is the sample size.
We employ a parameter-free distribution estimation framework where estimators are random distributions and utilize the Kullback-Leibler (KL) divergence as a loss function. Wu and Vos [J. Statist. Plann. Inference 142 (2012) 1525-1536] show that when an estimator obtained from an i.i.d. sample is viewed as a random distribution, the KL risk of the estimator decomposes in a fashion parallel to the mean squared error decomposition when the estimator is a real-valued random variable. In this paper, we explore how conditional versions of distribution expectation ($E^{\dagger}$) can be defined so th…
Different change-point type models encountered in statistical inference for stochastic processes give rise to different limiting likelihood ratio processes. In this paper we consider two such likelihood ratios. The first one is an exponential functional of a two-sided Poisson process driven by some parameter, while the second one is an exponential functional of a two-sided Brownian motion. We establish that for sufficiently small values of the parameter, the Poisson type likelihood ratio can be approximated by the Brownian type one. As a consequence, several statistically interesting quantitie…
We study holonomic gradient decent for maximum likelihood estimation of exponential-polynomial distribution, whose density is the exponential function of a polynomial in the random variable. We first consider the case that the support of the distribution is the set of positive reals. We show that the maximum likelihood estimate (MLE) can be easily computed by the holonomic gradient descent, even though the normalizing constant of this family does not have a closed-form expression and discuss determination of the degree of the polynomial based on the score test statistic. Then we present extens…
In many families of distributions, maximum likelihood estimation is intractable because the normalization constant for the density which enters into the likelihood function is not easily available. The score matching estimator of Hyvärinen (2005) provides an alternative where this normalization constant is not required. The corresponding estimating equations become linear for an exponential family. The score matching estimator is shown to be consistent and asymptotically normally distributed for such models, although not necessarily efficient. Gaussian linear concentration models are examples …
Stein's method of moment estimators for local dependency exponential random graph models
本文提出利用Stein估计器改进指数随机图模型的参数估计，通过Stein特征化方法获得新的估计器，解决传统最大似然估计计算复杂的问题。
Different change-point type models encountered in statistical inference for stochastic processes give rise to different limiting likelihood ratio processes. In a previous paper of one of the authors it was established that one of these likelihood ratios, which is an exponential functional of a two-sided Poisson process driven by some parameter, can be approximated (for sufficiently small values of the parameter) by another one, which is an exponential functional of a two-sided Brownian motion. In this paper we consider yet another likelihood ratio, which is the exponent of a two-sided compound…
Estimation of Tsallis entropy for exponentially distributed several populations
本文研究多个指数分布总体的Tsallis熵估计，提出Stein型改进估计和基于Brewster技术的光滑估计以降低风险，并给出贝叶斯估计及数值模拟验证。
In this note we prove the dual representation formula of the divergence between two distributions in a parametric model. Resulting estimators for the divergence as for the parameter are derived. These estimators do not make use of any grouping nor smoothing. It is proved that all differentiable divergences induce the same estimator of the parameter on any regular exponential family, which is nothing else but the MLE.
Fast computation and theoretical guarantees for the NPMLE in exponential family mixtures
提出数据压缩策略将NPMLE计算中似然评估成本降至样本量的多对数阶，并证明一类近似NPMLE的边际密度估计达到近乎参数收敛速度。
We prove the large deviation principle (LDP) for posterior distributions arising from subfamilies of full exponential families, allowing misspecification of the model. Moreover, motivated by the so-called inverse Sanov Theorem (see e.g. Ganesh and O'Connell 1999 and 2000), we prove the LDP for the corresponding maximum likelihood estimator, and we study the relationship between rate functions. In our setting, even in the non misspecified case, it is not true in general that the rate functions for posterior distributions and for maximum likelihood estimators are Kullback-Leibler divergences wit…
Novel closed-form point estimators for a weighted exponential family derived from likelihood equations
本文提出加权指数族的闭式点估计，并用自助法降低偏差，蒙特卡罗模拟验证其有效性。
Tsallis and Rényi entropies, which are monotone transformations of each other, are deformations of the celebrated Shannon entropy. Maximization of these deformed entropies, under suitable constraints, leads to the $q$-exponential family which has applications in non-extensive statistical physics, information theory and statistics. In previous information-geometric studies, the $q$-exponential family was analyzed using classical convex duality and Bregman divergence. In this paper, we show that a generalized $λ$-duality, where $λ= 1 - q$ is the constant information-geometric curvature, leads to…
We consider the problem of upper bounding the expected log-likelihood sub-optimality of the maximum likelihood estimate (MLE), or a conjugate maximum a posteriori (MAP) for an exponential family, in a non-asymptotic way. Surprisingly, we found no general solution to this problem in the literature. In particular, current theories do not hold for a Gaussian or in the interesting few samples regime. After exhibiting various facets of the problem, we show we can interpret the MAP as running stochastic mirror descent (SMD) on the log-likelihood. However, modern convergence results do not apply for …
While the asymptotic normality of the maximum likelihood estimator under regularity conditions is long established, this paper derives explicit bounds for the bounded Wasserstein distance between the distribution of the maximum likelihood estimator (MLE) and the normal distribution. For this task, we employ Stein's method. We focus on independent and identically distributed random variables, covering both discrete and continuous distributions as well as exponential and non-exponential families. In particular, a closed form expression of the MLE is not required. We also use a perturbation metho…
The exponential family is well known in machine learning and statistical physics as the maximum entropy distribution subject to a set of observed constraints, while the geometric mixture path is common in MCMC methods such as annealed importance sampling. Linking these two ideas, recent work has interpreted the geometric mixture path as an exponential family of distributions to analyze the thermodynamic variational objective (TVO). We extend these likelihood ratio exponential families to include solutions to rate-distortion (RD) optimization, the information bottleneck (IB) method, and recent …
An Accurate Standard Error Estimation for Quadratic Exponential Logistic Regressions by Applying Generalized Estimating Equations to Pseudo-Likelihoods
针对二次指数二元分布最大似然估计计算难、伪似然低估标准误的问题，本研究提出将广义估计方程应用于伪似然时采用独立工作相关结构可获一致估计，推导了理论性质并通过模拟与实例验证了方法有效性。
We consider the parameter estimation problem of a probabilistic generative model prescribed using a natural exponential family of distributions. For this problem, the typical maximum likelihood estimator usually overfits under limited training sample size, is sensitive to noise and may perform poorly on downstream predictive tasks. To mitigate these issues, we propose a distributionally robust maximum likelihood estimator that minimizes the worst-case expected log-loss uniformly over a parametric Kullback-Leibler ball around a parametric nominal distribution. Leveraging the analytical expressi…
Censored lifespans in a double-truncated sample: Maximum likelihood inference for the exponential distribution
针对双截断样本中因删失阻碍分析的问题，假设寿命服从指数分布并推导似然函数，对不可观测样本量进行轮廓化处理，证明了参数可识别性及 M-estimator 的一致性与渐近正态性，并以德国企业寿命为例验证。
The celebrated Bernstein von-Mises theorem ensures that credible regions from Bayesian posterior are well-calibrated when the model is correctly-specified, in the frequentist sense that their coverage probabilities tend to the nominal values as data accrue. However, this conventional Bayesian framework is known to lack robustness when the model is misspecified or only partly specified, such as in quantile regression, risk minimization based supervised/unsupervised learning and robust estimation. To overcome this difficulty, we propose a new Bayesian inferential approach that substitutes the (m…
Targeted maximum likelihood estimation (TMLE) is a general method for estimating parameters in semiparametric and nonparametric models. Each iteration of TMLE involves fitting a parametric submodel that targets the parameter of interest. We investigate the use of exponential families to define the parametric submodel. This implementation of TMLE gives a general approach for estimating any smooth parameter in the nonparametric model. A computational advantage of this approach is that each iteration of TMLE involves estimation of a parameter in an exponential family, which is a convex optimizati…
Online changepoint detection algorithms that are based on likelihood-ratio tests have been shown to have excellent statistical properties. However, a simple online implementation is computationally infeasible as, at time $T$, it involves considering $O(T)$ possible locations for the change. Recently, the FOCuS algorithm has been introduced for detecting changes in mean in Gaussian data that decreases the per-iteration cost to $O(\log T)$. This is possible by using pruning ideas, which reduce the set of changepoint locations that need to be considered at time $T$ to approximately $\log T$. We s…
Simple exponential smoothing is widely used in forecasting economic time series. This is because it is quick to compute and it generally delivers accurate forecasts. On the other hand, its multivariate version has received little attention due to the complications arising with the estimation. Indeed, standard multivariate maximum likelihood methods are affected by numerical convergence issues and bad complexity, growing with the dimensionality of the model. In this paper, we introduce a new estimation strategy for multivariate exponential smoothing, based on aggregating its observations into s…
It's all In the (Exponential) Family: An Equivalence between Maximum Likelihood Estimation and Control Variates for Sketching Algorithms
研究指数族中最大似然估计（MLE）与控制变量估计（CVE），证明特定条件下最优CVE与MLE渐近方差相同，给出MLE定点算法，实验显示该算法更快且数值稳定。
For a sample of Exponentially distributed durations we aim at point estimation and a confidence interval for its parameter. A duration is only observed if it has ended within a certain time interval, determined by a Uniform distribution. Hence, the data is a truncated empirical process that we can approximate by a Poisson process when only a small portion of the sample is observed, as is the case for our applications. We derive the likelihood from standard arguments for point processes, acknowledging the size of the latent sample as the second parameter, and derive the maximum likelihood estim…
The Maximum Likelihood Degree of Farlie Gumbel Morgenstern Bivariate Exponential Distribution
本文研究 Farlie-Gumbel-Morgenstern 二元指数分布参数的最大似然度，即对数似然方程在复数域上的解的个数。
Likelihood Geometry of the Gumbel's Type-I Bivariate Exponential Distribution
本文基于代数统计学，采用代数技术研究耿贝尔I型二元指数分布关联参数的最大似然度，明确其为复域上得分方程解的计重数数量。
Existence of maximum likelihood estimates in exponential random graph models
本文为指数随机图模型理论中的基础结论提供了简化证明，指出极大似然估计存在当且仅当目标统计量位于可实现统计量集合凸包的相对内部。
We demonstrate how to calculate posteriors for general CRM-based priors and likelihoods for Bayesian nonparametric models. We further show how to represent Bayesian nonparametric priors as a sequence of finite draws using a size-biasing approach---and how to represent full Bayesian nonparametric models via finite marginals. Motivated by conjugate priors based on exponential family representations of likelihoods, we introduce a notion of exponential families for CRMs, which we call exponential CRMs. This construction allows us to specify automatic Bayesian nonparametric conjugate priors for exp…
Likelihood inference of the non-stationary Hawkes process with non-exponential kernel
针对经典Hawkes过程的常数基线强度与指数激发核假设过于受限的问题，提出近似方法证明非平稳非指数核Hawkes过程的强度过程渐近遍历，以此推导似然函数的遍历极限，为其极大似然推断的大样本性质提供理论支撑。
A maximum likelihood algorithm for the estimation and renormalization of exponential densities
提出基于最大似然的算法，通过Levenberg-Marquardt优化求解矩匹配问题，实现指数密度的估计与重正化，并用于二维Ising模型临界温度计算。
Parameters defined via General Estimating Equations (GEE) can be estimated by maximizing the Empirical Likelihood (EL). Newey and Smith (2004) have recently shown that this EL estimator exhibits desirable higher-order asymptotic properties, namely, that its O(n^-1) bias is small and that bias-corrected EL is higher-order efficient. Although EL possesses these properties when the model is correctly specified, this paper shows that, in the presence of model misspecification, EL may cease to be root n convergent when the functions defining the moment conditions are unbounded (even when their expe…
We first consider the sequential detection of transient signals by generalizing the moving average chart to exponential family and study the false detection probability (FDP) and power of detection (POD) in the steady state. Then windowed adjusted signed (or modified directed) likelihood ratio chart is studied by treating it as normal random variable. In the multi-parameter exponential family, the detection of the transient change of one of the canonical parameters or a function of canonical parameters is considered by using the generalized adjusted signed likelihood ratio chart. Comparisons w…
Parameters defined via general estimating equations (GEE) can be estimated by maximizing the empirical likelihood (EL). Newey and Smith [Econometrica 72 (2004) 219--255] have recently shown that this EL estimator exhibits desirable higher-order asymptotic properties, namely, that its $O(n^{-1})$ bias is small and that bias-corrected EL is higher-order efficient. Although EL possesses these properties when the model is correctly specified, this paper shows that, in the presence of model misspecification, EL may cease to be root n convergent when the functions defining the moment conditions are …
Consider semiparametric models that display local asymptotic exponentiality (Ibragimov and Has'minskii (1981)), an asymptotic property of the likelihood associated with discontinuities of densities. Our interest goes to estimation of the location of such discontinuities while other aspects of the density form a nuisance parameter. It is shown that under certain conditions on model and prior, the posterior distribution displays Bernstein-von Mises-type asymptotic behaviour, with exponential distributions as the limiting sequence. In contrast to regular settings, the maximum likelihood estimator…
This thesis studies high-dimensional, continuous-valued pairwise Markov Random Fields. We are particularly interested in approximating pairwise densities whose logarithm belongs to a Sobolev space. For this problem we propose the method of exponential series [Crain, 1974; Barron and Sheu, 1991], which approximates the log density by a finite- dimensional exponential family with the number of sufficient statistics increasing with the sample size. We consider two approaches to estimating these models. The first is regularized maximum likelihood. This involves optimizing the sum of the log-likeli…
Concept of exponential family is generalized by simple and general exponential form. Simple and general potential are introduced. Maximum Entropy and Maximum Likelihood tasks are defined. ML task on the simple exponential form and ME task on the simple potentials are proved to be complementary in set-up and identical in solutions. ML task on the general exponential form and ME task on the general potentials are weakly complementary, leading to the same necessary conditions. A hypothesis about complementarity of ML and MiniMax Entropy tasks and identity of their solutions, brought up by a speci…
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。