We study holonomic gradient decent for maximum likelihood estimation of exponential-polynomial distribution, whose density is the exponential function of a polynomial in the random variable. We first consider the case that the support of the distribution is the set of positive reals. We show that the maximum likelihood estimate (MLE) can be easily computed by the holonomic gradient descent, even though the normalizing constant of this family does not have a closed-form expression and discuss determination of the degree of the polynomial based on the score test statistic. Then we present extensions to the whole real line and to the bivariate distribution on the positive orthant.
We propose an accelerated version of the holonomic gradient descent and apply it to calculating the maximum likelihood estimate (MLE) of the Fisher-Bingham distribution on a $d$-dimensional sphere. We derive a Pfaffian system (an integrable connection) and a series expansion associated with the normalizing constant with an error estimation. These enable us to solve some MLE problems up to dimension $d=7$ with a specified accuracy.
Recently, the school of Takemura and Takayama have developed a quite interesting minimization method called holonomic gradient descent method (HGD). It works by a mixed use of Pfaffian differential equation satisfied by an objective holonomic function and an iterative optimization method. They successfully applied the method to several maximum likelihood estimation (MLE) problems, which have been intractable in the past. On the other hand, in statistical models, it is not rare that parameters are constrained and therefore the MLE with constraints has been surely one of fundamental topics in st…
We give a new algorithm to find local maximum and minimum of a holonomic function and apply it for the Fisher-Bingham integral on the sphere $S^n$, which is used in the directional statistics. The method utilizes the theory and algorithms of holonomic systems.
Mila；Université de Montreal（蒙特利尔大学）；University of British Columbia（不列颠哥伦比亚大学）；Samsung（三星）；SAIT AI Lab（SAIT AI实验室）
We consider the problem of upper bounding the expected log-likelihood sub-optimality of the maximum likelihood estimate (MLE), or a conjugate maximum a posteriori (MAP) for an exponential family, in a non-asymptotic way. Surprisingly, we found no general solution to this problem in the literature. In particular, current theories do not hold for a Gaussian or in the interesting few samples regime. After exhibiting various facets of the problem, we show we can interpret the MAP as running stochastic mirror descent (SMD) on the log-likelihood. However, modern convergence results do not apply for …
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Duke University（杜克大学）
Information geometry applies concepts in differential geometry to probability and statistics and is especially useful for parameter estimation in exponential families where parameters are known to lie on a Riemannian manifold. Connections between the geometric properties of the induced manifold and statistical properties of the estimation problem are well-established. However developing first-order methods that scale to larger problems has been less of a focus in the information geometry community. The best known algorithm that incorporates manifold structure is the second-order natural gradie…
Amazon（亚马逊）；University of Oxford（牛津大学）；University of Pennsylvania（宾夕法尼亚大学）；Oxford University（牛津大学）
Methods for learning from data depend on various types of tuning parameters, such as penalization strength or step size. Since performance can depend strongly on these parameters, it is important to compare classes of estimators-by considering prescribed finite sets of tuning parameters-not just particularly tuned methods. In this work, we investigate classes of methods via the relative performance of the best method in the class. We consider the central problem of linear regression-with a random isotropic ground truth-and investigate the estimation performance of two fundamental methods, grad…
This paper studies early-stopped mirror descent applied to noisy sparse phase retrieval, which is the problem of recovering a $k$-sparse signal $\mathbf{x}^\star\in\mathbb{R}^n$ from a set of quadratic Gaussian measurements corrupted by sub-exponential noise. We consider the (non-convex) unregularized empirical risk minimization problem and show that early-stopped mirror descent, when equipped with the hyperbolic entropy mirror map and proper initialization, achieves a nearly minimax-optimal rate of convergence, provided the sample size is at least of order $k^2$ (modulo logarithmic term) and …
We propose an online parametric estimation method of stochastic differential equations with discrete observations and misspecified modelling based on online gradient descent. Our study provides uniform upper bounds for the risks of the estimators over a family of stochastic differential equations. The derivation of the bounds involves three underlying theoretical results: the analysis of the stochastic mirror descent algorithm based on dependent and biased subgradients, the simultaneous exponential ergodicity of classes of diffusion processes, and the proposal of loss functions whose approxima…
Optimal sampling for stochastic and natural gradient descent
Freie Universität Berlin（柏林自由大学）；Centrale Nantes & Nantes Université（南特中央理工学院与南特大学）；Laboratoire de Mathématiques Jean Leray, CNRS UMR 6629（让·勒雷数学实验室，法国国家科学研究中心）
Online simultaneous inference for quantiles via smoothed stochastic gradient descent
Department of Mathematics and Statistics（数学与统计学系）；Washington University in St.Louis（圣路易斯华盛顿大学）；Chair of Statistics（统计学教授职位）；Humboldt-Universität zu Berlin（柏林洪堡大学）；University of Chicago（芝加哥大学）
Riemannian Gradient Descent for Gaussian Mixture Models with unknown diagonal covariances
Centrale Lyon, INSA Lyon, Université Lyon 1, Université Jean Monnet, CNRS, ICJ UMR5208（里昂中央理工学院、里昂国立应用科学学院、里昂第一大学、让·莫内大学、法国国家科学研究中心、ICJ UMR5208）；Institut Universitaire de France (IUF)（法兰西学院）
