We study nonparametric maximum likelihood estimation of a log-concave probability density and its distribution and hazard function. Some general properties of these estimators are derived from two characterizations. It is shown that the rate of convergence with respect to supremum norm on a compact interval for the density and hazard rate estimator is at least $(\log(n)/n)^{1/3}$ and typically $(\log(n)/n)^{2/5}$, whereas the difference between the empirical and estimated distribution function vanishes with rate $o_{\mathrm{p}}(n^{-1/2})$ under certain regularity assumptions.
We study the smoothed log-concave maximum likelihood estimator of a probability distribution on $\mathbb{R}^d$. This is a fully automatic nonparametric density estimator, obtained as a canonical smoothing of the log-concave maximum likelihood estimator. We demonstrate its attractive features both through an analysis of its theoretical properties and a simulation study. Moreover, we use our methodology to develop a new test of log-concavity, and show how the estimator can be used as an intermediate stage of more involved procedures, such as constructing a classifier or estimating a functional o…
We present theoretical properties of the log-concave maximum likelihood estimator of a density based on an independent and identically distributed sample in $\mathbb{R}^d$. Our study covers both the case where the true underlying density is log-concave, and where this model is misspecified. We begin by showing that for a sequence of log-concave densities, convergence in distribution implies much stronger types of convergence -- in particular, it implies convergence in Hellinger distance and even in certain exponentially weighted total variation norms. In our main result, we prove the existence…
We propose a likelihood ratio statistic for forming hypothesis tests and confidence intervals for a nonparametrically estimated univariate regression function, based on the shape restriction of concavity (alternatively, convexity). Dealing with the likelihood ratio statistic requires studying an estimator satisfying a null hypothesis, that is, studying a concave least-squares estimator satisfying a further equality constraint. We study this null hypothesis least-squares estimator (NLSE) here, and use it to study our likelihood ratio statistic. The NLSE is the solution to a convex program, and …
We study nonparametric maximum likelihood estimation for two classes of multivariate distributions that imply strong forms of positive dependence; namely log-supermodular (MTP$_2$) distributions and log-$L^\#$-concave (LLC) distributions. In both cases we also assume log-concavity in order to ensure boundedness of the likelihood function. Given $n$ independent and identically distributed random vectors in $\mathbb R^d$ from one of our distributions, the maximum likelihood estimator (MLE) exists a.s. and is unique a.e. with probability one when $n\geq 3$. This holds independently of the ambient…
We study a likelihood ratio test for the location of the mode of a log-concave density. Our test is based on comparison of the log-likelihoods corresponding to the unconstrained maximum likelihood estimator of a log-concave density and the constrained maximum likelihood estimator where the constraint is that the mode of the density is fixed, say at $m$. The constrained estimation problem is studied in detail in Doss and Wellner [2018]. Here the results of that paper are used to show that, under the null hypothesis (and strict curvature of $-\log f$ at the mode), the likelihood ratio statistic …
The log-concave maximum likelihood estimator (MLE) problem answers: for a set of points $X_1,...X_n \in \mathbb R^d$, which log-concave density maximizes their likelihood? We present a characterization of the log-concave MLE that leads to an algorithm with runtime $poly(n,d, \frac 1 ε,r)$ to compute a log-concave distribution whose log-likelihood is at most $ε$ less than that of the MLE, and $r$ is parameter of the problem that is bounded by the $\ell_2$ norm of the vector of log-likelihoods the MLE evaluated at $X_1,...,X_n$.
This paper develops a general theory on rates of convergence of penalized spline estimators for function estimation when the likelihood functional is concave in candidate functions, where the likelihood is interpreted in a broad sense that includes conditional likelihood, quasi-likelihood, and pseudo-likelihood. The theory allows all feasible combinations of the spline degree, the penalty order, and the smoothness of the unknown functions. According to this theory, the asymptotic behaviors of the penalized spline estimators depends on interplay between the spline knot number and the penalty pa…
We consider nonparametric maximum-likelihood estimation of a log-concave density in case of interval-censored, right-censored and binned data. We allow for the possibility of a subprobability density with an additional mass at $+\infty$, which is estimated simultaneously. The existence of the estimator is proved under mild conditions and various theoretical aspects are given, such as certain shape and consistency properties. An EM algorithm is proposed for the approximate computation of the estimator and its performance is illustrated in two examples.
We study semiparametric time series models with innovations following a log-concave distribution. We propose a general maximum likelihood framework which allows us to estimate simultaneously the parameters of the model and the density of the innovations. This framework can be easily adapted to many well-known models, including ARMA, GARCH and ARMA-GARCH. Furthermore, we show that the estimator under our new framework is consistent in both ARMA and ARMA-GARCH settings. We demonstrate its finite sample performance via a thorough simulation study and apply it to model the daily log-return of FTSE…
We study the adaptation properties of the multivariate log-concave maximum likelihood estimator over three subclasses of log-concave densities. The first consists of densities with polyhedral support whose logarithms are piecewise affine. The complexity of such densities~$f$ can be measured in terms of the sum $Γ(f)$ of the numbers of facets of the subdomains in the polyhedral subdivision of the support induced by $f$. Given $n$ independent observations from a $d$-dimensional log-concave density with $d \in \{2,3\}$, we prove a sharp oracle inequality, which in particular implies that the Kull…
Phylogenetic trees are key data objects in biology, and the method of phylogenetic reconstruction has been highly developed. The space of phylogenetic trees is a nonpositively curved metric space. Recently, statistical methods to analyze the set of trees on this space are being developed utilizing this property. Meanwhile, in Euclidean space, the log-concave maximum likelihood method has emerged as a new nonparametric method for probability density estimation. In this paper, we derive a sufficient condition for the existence and uniqueness of the log-concave maximum likelihood estimator on tre…
The Ohio State University（俄亥俄州立大学）；University of Southern California（南加州大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
We study the problem of learning multivariate log-concave densities with respect to a global loss function. We obtain the first upper bound on the sample complexity of the maximum likelihood estimator (MLE) for a log-concave density on $\mathbb{R}^d$, for all $d \geq 4$. Prior to this work, no finite sample upper bound was known for this estimator in more than $3$ dimensions. In more detail, we prove that for any $d \geq 1$ and $ε>0$, given $\tilde{O}_d((1/ε)^{(d+3)/2})$ samples drawn from an unknown log-concave density $f_0$ on $\mathbb{R}^d$, the MLE outputs a hypothesis $h$ that with high p…
We present a new approach for inference about a log-concave distribution: Instead of using the method of maximum likelihood, we propose to incorporate the log-concavity constraint in an appropriate nonparametric confidence set for the cdf $F$. This approach has the advantage that it automatically provides a measure of statistical uncertainty and it thus overcomes a marked limitation of the maximum likelihood estimate. In particular, we show how to construct confidence bands for the density that have a finite sample guaranteed confidence level. The nonparametric confidence set for $F$ which we …
We propose a unified framework for likelihood-based regression modeling when the response variable has finite support. Our work is motivated by the fact that, in practice, observed data are discrete and bounded. The proposed methods assume a model which includes models previously considered for interval-censored variables with log-concave distributions as special cases. The resulting log-likelihood is concave, which we use to establish asymptotic normality of its maximizer as the number of observations $n$ tends to infinity with the number of parameters $d$ fixed, and rates of convergence of $…
In this paper, we study two problems: (1) estimation of a $d$-dimensional log-concave distribution and (2) bounded multivariate convex regression with random design with an underlying log-concave density or a compactly supported distribution with a continuous density. First, we show that for all $d \ge 4$ the maximum likelihood estimators of both problems achieve an optimal risk of $Θ_d(n^{-2/(d+1)})$ (up to a logarithmic factor) in terms of squared Hellinger distance and $L_2$ squared distance, respectively. Previously, the optimality of both these estimators was known only for $d\le 3$. We a…
Let X_1, ..., X_n be independent and identically distributed random vectors with a log-concave (Lebesgue) density f. We first prove that, with probability one, there exists a unique maximum likelihood estimator of f. The use of this estimator is attractive because, unlike kernel density estimation, the method is fully automatic, with no smoothing parameters to choose. Although the existence proof is non-constructive, we are able to reformulate the issue of computation in terms of a non-differentiable convex optimisation problem, and thus combine techniques of computational geometry with Shor's…
We establish global rates of convergence for the Maximum Likelihood Estimators (MLEs) of log-concave and $s$-concave densities on $\mathbb{R}$. The main finding is that the rate of convergence of the MLE in the Hellinger metric is no worse than $n^{-2/5}$ when $-1 < s < \infty$ where $s=0$ corresponds to the log-concave case. We also show that the MLE does not exist for the classes of $s$-concave densities with $s < - 1$.
Motivated by studies in biological sciences to detect differentially expressed genes, a semiparametric two-component mixture model with one known component is being studied in this paper. Assuming the density of the unknown component to be log-concave, which contains a very broad family of densities, we develop a semiparametric maximum likelihood estimator and propose an EM algorithm to compute it. Our new estimation method finds the mixing proportions and the distribution of the unknown component simultaneously. We establish the identifiability of the proposed semiparametric mixture model and…
Maximum likelihood estimation of a log-concave probability density is formulated as a convex optimization problem and shown to have an equivalent dual formulation as a constrained maximum Shannon entropy problem. Closely related maximum Renyi entropy estimators that impose weaker concavity restrictions on the fitted density are also considered, notably a minimum Hellinger discrepancy estimator that constrains the reciprocal of the square-root of the density to be concave. A limiting form of these estimators constrains solutions to the class of quasi-concave densities.
We study nonparametric maximum likelihood estimation of a log-concave density function $f_0$ which is known to satisfy further constraints, where either (a) the mode $m$ of $f_0$ is known, or (b) $f_0$ is known to be symmetric about a fixed point $m$. We develop asymptotic theory for both constrained log-concave maximum likelihood estimators (MLE's), including consistency, global rates of convergence, and local limit distribution theory. In both cases, we find the MLE's pointwise limit distribution at $m$ (either the known mode or the known center of symmetry) and at a point $x_0 \ne m$. Softw…
In recent years, log-concave density estimation via maximum likelihood estimation has emerged as a fascinating alternative to traditional nonparametric smoothing techniques, such as kernel density estimation, which require the choice of one or more bandwidths. The purpose of this article is to describe some of the properties of the class of log-concave densities on $\mathbb{R}^d$ which make it so attractive from a statistical perspective, and to outline the latest methodological, theoretical and computational advances in the area.
In this paper, we study the approximation and estimation of $s$-concave densities via Rényi divergence. We first show that the approximation of a probability measure $Q$ by an $s$-concave densities exists and is unique via the procedure of minimizing a divergence functional proposed by Koenker and Mizera (2010) if and only if $Q$ admits full-dimensional support and a first moment. We also show continuity of the divergence functional in $Q$: if $Q_n \to Q$ in the Wasserstein metric, then the projected densities converge in weighted $L_1$ metrics and uniformly on closed subsets of the continuity…
The log-concave maximum likelihood estimator of a density on the real line based on a sample of size $n$ is known to attain the minimax optimal rate of convergence of $O(n^{-4/5})$ with respect to, e.g., squared Hellinger distance. In this paper, we show that it also enjoys attractive adaptation properties, in the sense that it achieves a faster rate of convergence when the logarithm of the true density is $k$-affine (i.e.\ made up of $k$ affine pieces), provided $k$ is not too large. Our results use two different techniques: the first relies on a new Marshall's inequality for log-concave dens…
The most widely used technology to identify the proteins present in a complex biological sample is tandem mass spectrometry, which quickly produces a large collection of spectra representative of the peptides (i.e., protein subsequences) present in the original sample. In this work, we greatly expand the parameter learning capabilities of a dynamic Bayesian network (DBN) peptide-scoring algorithm, Didea, by deriving emission distributions for which its conditional log-likelihood scoring function remains concave. We show that this class of emission distributions, called Convex Virtual Emissions…
The estimation of a log-concave density on $\mathbb{R}$ is a canonical problem in the area of shape-constrained nonparametric inference. We present a Bayesian nonparametric approach to this problem based on an exponentiated Dirichlet process mixture prior and show that the posterior distribution converges to the log-concave truth at the (near-) minimax rate in Hellinger distance. Our proof proceeds by establishing a general contraction result based on the log-concave maximum likelihood estimator that prevents the need for further metric entropy calculations. We also present two computationally…
We consider two nonparametric procedures for estimating a concave distribution function based on data corrupted with additive noise generated by a bounded decreasing density on $(0,\infty)$. For the maximum likelihood (ML) estimator and least squares (LS) estimator, we state qualitative properties, prove consistency and propose a computational algorithm. For the LS estimator and its derivative, we also derive the pointwise asymptotic distribution. Moreover, the rate $n^{-2/5}$ achieved by the LS estimator is shown to be minimax for estimating the distribution function at a fixed point.
We study the approximation of arbitrary distributions $P$ on $d$-dimensional space by distributions with log-concave density. Approximation means minimizing a Kullback--Leibler-type functional. We show that such an approximation exists if and only if $P$ has finite first moments and is not supported by some hyperplane. Furthermore we show that this approximation depends continuously on $P$ with respect to Mallows distance $D_1(\cdot,\cdot)$. This result implies consistency of the maximum likelihood estimator of a log-concave density under fairly general conditions. It also allows us to prove e…
The estimation of a log-concave density on $\mathbb{R}^d$ represents a central problem in the area of nonparametric inference under shape constraints. In this paper, we study the performance of log-concave density estimators with respect to global loss functions, and adopt a minimax approach. We first show that no statistical procedure based on a sample of size $n$ can estimate a log-concave density with respect to the squared Hellinger loss function with supremum risk smaller than order $n^{-4/5}$, when $d=1$, and order $n^{-2/(d+1)}$ when $d \geq 2$. In particular, this reveals a sense in wh…
We study local asymptotic normality of M-estimates of convex minimization in an infinite dimensional parameter space. The objective function of M-estimates is not necessary differentiable and is possibly subject to convex constraints. In the above circumstance, narrow convergence with respect to uniform convergence fails to hold, because of the strength of it's topology. A new approach we propose to the lack-of-uniform-convergence is based on Mosco-convergence that is weaker topology than uniform convergence. By applying narrow convergence with respect to Mosco topology, we develop an infinite…
In this article, we revisit the problem of fitting a mixture model under the assumption that the mixture components are symmetric and log-concave. To this end, we first study the nonparametric maximum likelihood estimation (NPMLE) of a monotone log-concave probability density. By following the arguments of Rufibach (2006), we show that the NPMLE is uniformly consistent with respect to the supremum norm on compact subsets of the interior of the support. To fit the mixture model, we propose a semiparametric EM (SEM) algorithm, which can be adapted to other semiparametric mixture models. In our n…
A novel computational approach to log-concave density estimation is proposed. Previous approaches utilize the piecewise-affine parametrization of the density induced by the given sample set. The number of parameters as well as non-smooth subgradient-based convex optimization for determining the maximum likelihood density estimate cause long runtimes for dimensions $d \geq 2$ and large sample sets. The presented approach is based on mildly non-convex smooth approximations of the objective function and \textit{sparse}, adaptive piecewise-affine density parametrization. Established memory-efficie…
Reduced-rank regression is a dimensionality reduction method with many applications. The asymptotic theory for reduced rank estimators of parameter matrices in multivariate linear models has been studied extensively. In contrast, few theoretical results are available for reduced-rank multivariate generalised linear models. We develop M-estimation theory for concave criterion functions that are maximised over parameters spaces that are neither convex nor closed. These results are used to derive the consistency and asymptotic distribution of maximum likelihood estimators in reduced-rank multivar…
We revisit the problem of estimating the center of symmetry $θ$ of an unknown symmetric density $f$. Although Stone (1975), Van Eden (1970), and Sacks (1975) constructed adaptive estimators of $θ$ in this model, their estimators depend on tuning parameters. In an effort to circumvent the dependence on tuning parameters, we impose an additional assumption of log-concavity on $f$. We show that in this shape-restricted model, the maximum likelihood estimator (MLE) of $θ$ exists. We also study some truncated one-step estimators and show that they are $\sqrt{n}-$consistent, and nearly achieve the a…
Independent Component Analysis (ICA) models are very popular semiparametric models in which we observe independent copies of a random vector $X = AS$, where $A$ is a non-singular matrix and $S$ has independent components. We propose a new way of estimating the unmixing matrix $W = A^{-1}$ and the marginal distributions of the components of $S$ using nonparametric maximum likelihood. Specifically, we study the projection of the empirical distribution onto the subset of ICA distributions having log-concave marginals. We show that, from the point of view of estimating the unmixing matrix, it make…
The paper presents a novel asymptotic distribution for a mle when the log--likelihood is strictly concave in the parameter for all data points; for example, the exponential family. The new asymptotic distribution can be seen as a refinement of the usual normal asymptotic distribution and is comparable to an Edgeworth expansion. However, it is obtained with weaker conditions than even those for asymptotic normality. The same technique is then used to find the exact distribution of the weighted likelihood bootstrap sampler.
In Statistics, log-concave density estimation is a central problem within the field of nonparametric inference under shape constraints. Despite great progress in recent years on the statistical theory of the canonical estimator, namely the log-concave maximum likelihood estimator, adoption of this method has been hampered by the complexities of the non-smooth convex optimization problem that underpins its computation. We provide enhanced understanding of the structural properties of this optimization problem, which motivates the proposal of new algorithms, based on both randomized and Nesterov…
Determinantal point processes (DPPs) have wide-ranging applications in machine learning, where they are used to enforce the notion of diversity in subset selection problems. Many estimators have been proposed, but surprisingly the basic properties of the maximum likelihood estimator (MLE) have received little attention. The difficulty is that it is a non-concave maximization problem, and such functions are notoriously difficult to understand in high dimensions, despite their importance in modern machine learning. Here we study both the local and global geometry of the expected log-likelihood f…
Shape-constrained density estimation is an important topic in mathematical statistics. We focus on densities on $\mathbb{R}^d$ that are log-concave, and we study geometric properties of the maximum likelihood estimator (MLE) for weighted samples. Cule, Samworth, and Stewart showed that the logarithm of the optimal log-concave density is piecewise linear and supported on a regular subdivision of the samples. This defines a map from the space of weights to the set of regular subdivisions of the samples, i.e. the face poset of their secondary polytope. We prove that this map is surjective. In fac…
We study estimation of multivariate densities $p$ of the form $p(x)=h(g(x))$ for $x\in \mathbb {R}^d$ and for a fixed monotone function $h$ and an unknown convex function $g$. The canonical example is $h(y)=e^{-y}$ for $y\in \mathbb {R}$; in this case, the resulting class of densities [\mathcal {P}(e^{-y})={p=\exp(-g):g is convex}] is well known as the class of log-concave densities. Other functions $h$ allow for classes of densities with heavier tails than the log-concave class. We first investigate when the maximum likelihood estimator $\hat{p}$ exists for the class $\mathcal {P}(h)$ for var…
In this article, we revisit the problem of estimating the unknown zero-symmetric distribution in a two-component location mixture model, considered in previous works, now under the assumption that the zero-symmetric distribution has a log-concave density. When consistent estimators for the shift locations and mixing probability are used, we show that the nonparametric log-concave Maximum Likelihood estimator (MLE) of both the mixed density and that of the unknown zero-symmetric component are consistent in the Hellinger distance. In case the estimators for the shift locations and mixing probabi…
Let $\{P_θ:θ\in {\mathbb R}^d\}$ be a log-concave location family with $P_θ(dx)=e^{-V(x-θ)}dx,$ where $V:{\mathbb R}^d\mapsto {\mathbb R}$ is a known convex function and let $X_1,\dots, X_n$ be i.i.d. r.v. sampled from distribution $P_θ$ with an unknown location parameter $θ.$ The goal is to estimate the value $f(θ)$ of a smooth functional $f:{\mathbb R}^d\mapsto {\mathbb R}$ based on observations $X_1,\dots, X_n.$ In the case when $V$ is sufficiently smooth and $f$ is a functional from a ball in a Hölder space $C^s,$ we develop estimators of $f(θ)$ with minimax optimal error rates measured by…
For Markov random fields on $\mathbb{Z}^d$ with finite state space, we address the statistical estimation of the basic neighborhood, the smallest region that determines the conditional distribution at a site on the condition that the values at all other sites are given. A modification of the Bayesian Information Criterion, replacing likelihood by pseudo-likelihood, is proved to provide strongly consistent estimation from observing a realization of the field on increasing finite regions: the estimated basic neighborhood equals the true one eventually almost surely, not assuming any prior bound …
Penalized likelihood methods are fundamental to ultra-high dimensional variable selection. How high dimensionality such methods can handle remains largely unknown. In this paper, we show that in the context of generalized linear models, such methods possess model selection consistency with oracle properties even for dimensionality of Non-Polynomial (NP) order of sample size, for a class of penalized likelihood approaches using folded-concave penalty functions, which were introduced to ameliorate the bias problems of convex penalty functions. This fills a long-standing gap in the literature whe…
Statistical models with latent structure have a history going back to the 1950s and have seen widespread use in the social sciences and, more recently, in computational biology and in machine learning. Here we study the basic latent class model proposed originally by the sociologist Paul F. Lazarfeld for categorical variables, and we explain its geometric structure. We draw parallels between the statistical and geometric properties of latent class models and we illustrate geometrically the causes of many problems associated with maximum likelihood estimation and related statistical inference. …
We propose a novel and flexible rank-breaking-then-composite-marginal-likelihood (RBCML) framework for learning random utility models (RUMs), which include the Plackett-Luce model. We characterize conditions for the objective function of RBCML to be strictly log-concave by proving that strict log-concavity is preserved under convolution and marginalization. We characterize necessary and sufficient conditions for RBCML to satisfy consistency and asymptotic normality. Experiments on synthetic data show that RBCML for Gaussian RUMs achieves better statistical efficiency and computational efficien…
For the basic maximum likelihood estimating function of the two parameters Weibull distribution, a simple proof on its global monotonicity is given to ensure the existence and uniqueness of its solution. The boundary of the function's first-order derivative is defined based on its scale-free property. With a bounded derivative, the possible range of the root of this function can be determined. A novel root-finding algorithm employing these established results is proposed accordingly, its convergence is proved analytically as well. Compared with other typical algorithms for this problem, the ef…
The saddlepoint approximation gives an approximation to the density of a random variable in terms of its moment generating function. When the underlying random variable is itself the sum of $n$ unobserved i.i.d. terms, the basic classical result is that the relative error in the density is of order $1/n$. If instead the approximation is interpreted as a likelihood and maximised as a function of model parameters, the result is an approximation to the maximum likelihood estimate (MLE) that can be much faster to compute than the true MLE. This paper proves the analogous basic result for the appro…
